LVM: One or more devices used as PVs in VG have changed sizes

liv913 asked:

I recently installed LVM on Ubuntu 18.04 running on a physical workstation, and then I created my first LV. After first reboot I found a problem, as I cannot activate it anymore.

I noticed this error:

[email protected]:/var/log# pvdisplay -v
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
  WARNING: Device /dev/sdb has size of 312573551 sectors which is smaller than corresponding PV size of 312581808 sectors. Was device resized?
  One or more devices used as PVs in VG backup_vg have changed sizes.
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               backup_vg
  PV Size               149,05 GiB / not usable <3,84 MiB
  Allocatable           yes (but full)
  PE Size               4,00 MiB
  Total PE              38156
  Free PE               0
  Allocated PE          38156
  PV UUID               LHDgmj-QGCd-xJnF-QhqH-B61a-iYnd-BpN2ZI

  --- Physical volume ---
  PV Name               /dev/sdd
  VG Name               backup_vg
  PV Size               149,05 GiB / not usable <3,84 MiB
  Allocatable           yes (but full)
  PE Size               4,00 MiB
  Total PE              38156
  Free PE               0
  Allocated PE          38156
  PV UUID               lXbczN-5bXe-C2lB-2Wq1-khbE-19SO-O3aHdI

Here there are some display commands for VG and PV:

[email protected]:/media# vgdisplay 
  WARNING: Device /dev/sdb has size of 312573551 sectors which is smaller than corresponding PV size of 312581808 sectors. Was device resized?
  One or more devices used as PVs in VG backup_vg have changed sizes.
  --- Volume group ---
  VG Name               backup_vg
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               298,09 GiB
  PE Size               4,00 MiB
  Total PE              76312
  Alloc PE / Size       76312 / 298,09 GiB
  Free  PE / Size       0 / 0   
  VG UUID               lDCvsV-fOvb-qFWM-hjqF-9xbo-TClo-e0Vzca

[email protected]:/var/log# lvdisplay 
  WARNING: Device /dev/sdb has size of 312573551 sectors which is smaller than corresponding PV size of 312581808 sectors. Was device resized?
  One or more devices used as PVs in VG backup_vg have changed sizes.
  --- Logical volume ---
  LV Path                /dev/backup_vg/backup
  LV Name                backup
  VG Name                backup_vg
  LV UUID                lu8Jqs-7AD3-KNTH-2NP2-MQfP-zoxr-Ax2rPF
  LV Write Access        read/write
  LV Creation host, time lvrome, 2018-08-29 20:34:43 +0200
  LV Status              suspended
  # open                 0
  LV Size                298,09 GiB
  Current LE             76312
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

I tried to force activation for VG and then for LV, but I cannot go on:

[email protected]:/media# vgchange -ay
  WARNING: Device /dev/sdb has size of 312573551 sectors which is smaller than corresponding PV size of 312581808 sectors. Was device resized?
  One or more devices used as PVs in VG backup_vg have changed sizes.
  device-mapper: resume ioctl on  (253:0) failed: Argomento non valido
  Unable to resume backup_vg-backup (253:0)
  1 logical volume(s) in volume group "backup_vg" now active
[email protected]:/media# lvchange -ay backup_vg
  WARNING: Device /dev/sdb has size of 312573551 sectors which is smaller than corresponding PV size of 312581808 sectors. Was device resized?
  One or more devices used as PVs in VG backup_vg have changed sizes.
  device-mapper: resume ioctl on  (253:0) failed: Argomento non valido
  Unable to resume backup_vg-backup (253:0)

I’m sure I haven’t resized PVs after installation. By the way, I created them through pvcreate /dev/sdb /dev/sdd, vgcreate backup_vg /dev/sdb /dev/sdd, lvcreate --type striped -i 2 -l 100%FREE -n backup backup_vg.

Here’s the output from lsblk:

[email protected]:~# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 223,6G  0 disk 
├─sda1   8:1    0    50G  0 part 
├─sda3   8:3    0    40G  0 part /
├─sda4   8:4    0     1K  0 part 
├─sda5   8:5    0     4G  0 part [SWAP]
├─sda6   8:6    0    50G  0 part /home
└─sda7   8:7    0  79,3G  0 part /media/Documenti
sdb      8:16   0   149G  0 disk 
sdc      8:32   0 465,8G  0 disk 
└─sdc1   8:33   0 465,8G  0 part /media/Dati
sdd      8:48   0 149,1G  0 disk 
sde      8:64   0 465,7G  0 disk 
└─sde1   8:65   0 465,7G  0 part 
sr0     11:0    1  1024M  0 rom  
sr1     11:1    1  1024M  0 rom  
sr2     11:2    1    30M  0 rom  

SMART status for disks:

/dev/sdb

[email protected]:~# hdparm -I /dev/sdb

/dev/sdb:

ATA device, with non-removable media
    Model Number:       ST3160813AS                             
    Serial Number:      
    Firmware Revision:  CC2H    
    Transport:          Serial
Standards:
    Used: unknown (minor revision code 0x0029) 
    Supported: 8 7 6 5 
    Likely used: 8
Configuration:
    Logical     max current
    cylinders   16383   16383
    heads       16  16
    sectors/track   63  63
    --
    CHS current addressable sectors:    16514064
    LBA    user addressable sectors:   268435455
    LBA48  user addressable sectors:   312573551
    Logical/Physical Sector size:           512 bytes
    device size with M = 1024*1024:      152623 MBytes
    device size with M = 1000*1000:      160037 MBytes (160 GB)
    cache/buffer size  = 8192 KBytes
    Nominal Media Rotation Rate: 7200
Capabilities:
    LBA, IORDY(can be disabled)
    Queue depth: 32
    Standby timer values: spec'd by Standard, no device specific minimum
    R/W multiple sector transfer: Max = 16  Current = 16
    Recommended acoustic management value: 208, current value: 208
    DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
         Cycle time: min=120ns recommended=120ns
    PIO: pio0 pio1 pio2 pio3 pio4 
         Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
    Enabled Supported:
       *    SMART feature set
            Security Mode feature set
       *    Power Management feature set
       *    Write cache
       *    Look-ahead
       *    Host Protected Area feature set
       *    WRITE_BUFFER command
       *    READ_BUFFER command
       *    DOWNLOAD_MICROCODE
            SET_MAX security extension
       *    Automatic Acoustic Management feature set
       *    48-bit Address feature set
       *    Device Configuration Overlay feature set
       *    Mandatory FLUSH_CACHE
       *    FLUSH_CACHE_EXT
       *    SMART error logging
       *    SMART self-test
       *    General Purpose Logging feature set
       *    WRITE_{DMA|MULTIPLE}_FUA_EXT
       *    64-bit World wide name
            Write-Read-Verify feature set
       *    WRITE_UNCORRECTABLE_EXT command
       *    {READ,WRITE}_DMA_EXT_GPL commands
       *    Segmented DOWNLOAD_MICROCODE
       *    Gen1 signaling speed (1.5Gb/s)
       *    Gen2 signaling speed (3.0Gb/s)
       *    Native Command Queueing (NCQ)
       *    Phy event counters
            Device-initiated interface power management
       *    Software settings preservation
       *    SMART Command Transport (SCT) feature set
       *    SCT Read/Write Long (AC1), obsolete
       *    SCT Write Same (AC2)
       *    SCT Error Recovery Control (AC3)
       *    SCT Features Control (AC4)
       *    SCT Data Tables (AC5)
            unknown 206[12] (vendor specific)
Security: 
    Master password revision code = 65534
        supported
    not enabled
    not locked
    not frozen
    not expired: security count
        supported: enhanced erase
    28min for SECURITY ERASE UNIT. 28min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 5000c50015125925
    NAA     : 5
    IEEE OUI    : 000c50
    Unique ID   : 015125925
Checksum: correct

[email protected]:~# smartctl -H /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-33-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

/dev/sdd

[email protected]:~# hdparm -I /dev/sdd

/dev/sdd:

ATA device, with non-removable media
    Model Number:       ST3160813AS                             
    Serial Number:      
    Firmware Revision:  CC2H    
    Transport:          Serial
Standards:
    Used: unknown (minor revision code 0x0029) 
    Supported: 8 7 6 5 
    Likely used: 8
Configuration:
    Logical     max current
    cylinders   16383   16383
    heads       16  16
    sectors/track   63  63
    --
    CHS current addressable sectors:    16514064
    LBA    user addressable sectors:   268435455
    LBA48  user addressable sectors:   312581808
    Logical/Physical Sector size:           512 bytes
    device size with M = 1024*1024:      152627 MBytes
    device size with M = 1000*1000:      160041 MBytes (160 GB)
    cache/buffer size  = 8192 KBytes
    Nominal Media Rotation Rate: 7200
Capabilities:
    LBA, IORDY(can be disabled)
    Queue depth: 32
    Standby timer values: spec'd by Standard, no device specific minimum
    R/W multiple sector transfer: Max = 16  Current = 16
    Recommended acoustic management value: 208, current value: 254
    DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
         Cycle time: min=120ns recommended=120ns
    PIO: pio0 pio1 pio2 pio3 pio4 
         Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
    Enabled Supported:
       *    SMART feature set
            Security Mode feature set
       *    Power Management feature set
       *    Write cache
       *    Look-ahead
       *    Host Protected Area feature set
       *    WRITE_BUFFER command
       *    READ_BUFFER command
       *    DOWNLOAD_MICROCODE
            SET_MAX security extension
       *    Automatic Acoustic Management feature set
       *    48-bit Address feature set
       *    Device Configuration Overlay feature set
       *    Mandatory FLUSH_CACHE
       *    FLUSH_CACHE_EXT
       *    SMART error logging
       *    SMART self-test
       *    General Purpose Logging feature set
       *    WRITE_{DMA|MULTIPLE}_FUA_EXT
       *    64-bit World wide name
            Write-Read-Verify feature set
       *    WRITE_UNCORRECTABLE_EXT command
       *    {READ,WRITE}_DMA_EXT_GPL commands
       *    Segmented DOWNLOAD_MICROCODE
       *    Gen1 signaling speed (1.5Gb/s)
       *    Gen2 signaling speed (3.0Gb/s)
       *    Native Command Queueing (NCQ)
       *    Phy event counters
            Device-initiated interface power management
       *    Software settings preservation
       *    SMART Command Transport (SCT) feature set
       *    SCT Read/Write Long (AC1), obsolete
       *    SCT Write Same (AC2)
       *    SCT Error Recovery Control (AC3)
       *    SCT Features Control (AC4)
       *    SCT Data Tables (AC5)
            unknown 206[12] (vendor specific)
Security: 
    Master password revision code = 65534
        supported
    not enabled
    not locked
    not frozen
    not expired: security count
        supported: enhanced erase
    32min for SECURITY ERASE UNIT. 32min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 5000c5001507abcc
    NAA     : 5
    IEEE OUI    : 000c50
    Unique ID   : 01507abcc
Checksum: correct

[email protected]:~# smartctl -H /dev/sdd
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-33-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

That’s my LVM version:

[email protected]:/var/log# lvm version
  LVM version:     2.02.176(2) (2017-11-03)
  Library version: 1.02.145 (2017-11-03)
  Driver version:  4.37.0
  Configuration:   ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-clvmd=corosync --with-cluster=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service --enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync

Can you help me to find a solution?

My answer:


The order that disks are enumerated is not persistent between reboots. So /dev/sdb and /dev/sdd could be reversed each time you reboot the system. Or switched with /dev/sda or /dev/sdc. Or something else.

If you are doing something that depends on referring to the same physical disk each time the system boots, it’s best to use the symbolic links in /dev/disk/by-id as Linux makes sure these always point to the same physical disks, even if their /dev/sd* specials change.

For example:

pvcreate /dev/disk/by-id/ata-SEAGATE-ST3160813AS_*whatever # you hid the serial number

Technically this shouldn’t be necessary with LVM as it uses its own UUIDs internally for everything, at least after the PVs are created, but it’s a good habit to get into.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.