Debian DRBD: Tips, Tricks and Commands

Here you can find some common commands related to DRBD. They will be added once I find use of them on my daily work. Stay tuned!

How to force re-sync by invalidating the data on the node you are in:

root@eave:~# [color=green]drbdadm invalidate nfs[/color]
root@eave:~# cat /proc/drbd 
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9 
 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:28860 dw:24380 dr:0 al:0 bm:2 lo:36 pe:434 ua:35 ap:0 ep:1 wo:f oos:26192512
	[>....................] sync'ed:  0.2% (25576/25600)Mfinish: 0:19:50 speed: 21,888 (21,888) want: 51,200 K/sec

How to force re-sync by invalidating the data on the other node:

root@eave:~# [color=green]drbdadm invalidate_remote nfs[/color]
root@eave:~# cat /proc/drbd 
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9 
 1: cs:SyncTarget ro:Secondary/Primary ds:UpToDate/Inconsistent C r-----
    ns:0 nr:6073892 dw:6066212 dr:0 al:0 bm:370 lo:61 pe:7454 ua:60 ap:0 ep:1 wo:f oos:20155504
	[===>................] sync'ed: 23.2% (19680/25600)Mfinish: 0:24:42 speed: 13,596 (11,220) want: 51,200 K/sec

How to fix StandAlone Unknown drbd resource:

1. On primary node:

drbdadm connect nfs

2. On secondary node:

drbdadm -- --discard-my-data connect nfs

Example:

root@eave:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: 00000000000000000000000 
 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r-----
    ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:524032
    
root@eave:~# drbdadm connect nfs

root@candy:~# drbdadm -- --discard-my-data connect nfs

root@eave:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: 00000000000000000000000 
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:524032 nr:0 dw:0 dr:525028 al:0 bm:32 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Split Brain Recovery

:

How to get pass the WFConnection state

This happens usually when the network connection between nodes was cut off (split brain). In this situation, if you do not have drbd configured to recover automatically, you have to step in. Version 8.3: On secondary node (where the data will be discarded):

drbdadm down nfs

drbdadm -- --discard-my-data connect nfs

drbdadm attach nfs

Example:

root@candy:~# drbdadm down nfs
root@candy:~# cat /proc/drbd 
version: 8.3.11 (api:88/proto:86-96)
srcversion: 00000000000000000000000 
 0: cs:Unconfigured

root@candy:~# drbdadm -- --discard-my-data connect nfs
root@candy:~# cat /proc/drbd 
version: 8.3.11 (api:88/proto:86-96)
srcversion: 00000000000000000000000 
 0: cs:WFConnection ro:Secondary/Unknown ds:Diskless/DUnknown C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

root@candy:~# drbdadm attach nfs
root@candy:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: 00000000000000000000000 
 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:66680 dw:58476 dr:0 al:0 bm:126 lo:146 pe:547 ua:145 ap:0 ep:1 wo:f oos:66432
	[========>...........] sync'ed: 48.4% (66432/124908)K
	finish: 0:00:11 speed: 5,844 (5,844) want: 51,200 K/sec

For version 8.4: Secondary node:

drbdadm disconnect nfs

drbdadm -- --discard-my-data connect nfs

Primary node (if not connected):

drbdadm connect nfs

How to fix uuid_compare()=-1000 by rule 100

This happens usually after a split brain and normally this error comes with the explanation. However, in this situation I got only uuid_compare()=-1000 by rule 100 with nothing else. The fix here was to recreate the meta device:

# drbdadm detach nfs

# drbdadm create-md nfs

# drbdadm up nfs

Example:

root@eave:~# drbdadm detach nfs
root@eave:~# drbdadm create-md nfs
md_offset 0
al_offset 4096
bm_offset 36864

Found some data

 ==> This might destroy existing data! <==

Do you want to proceed?
[need to type 'yes' to confirm] yes


Valid meta data seems to be in place.
Do you really want to overwrite?
[need to type 'yes' to confirm] yes

Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
root@eave:~# drbdadm up nfs
root@eave:~# cat /proc/drbd 
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9 
 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate B r-----
    ns:0 nr:748024 dw:716024 dr:0 al:0 bm:42 lo:252 pe:4168 ua:250 ap:0 ep:1 wo:d oos:72684800
	[>....................] sync'ed:  1.0% (70980/71680)Mfinish: 1:15:55 speed: 15,936 (13,760) want: 30,720 K/sec

 2: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate B r-----
    ns:0 nr:27316 dw:27316 dr:0 al:0 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
 3: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate B r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
 4: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate B r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0

How to populate the metadata device and startup drbd:

drbdadm create-md nfs

drbdadm up all

cat /proc/drbd

Example:

root@candy:~# drbdadm create-md nfs
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
root@candy:~# drbdadm up all
root@candy:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: 00000000000000000000000 
 0: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:26214144

How to start the first sync of drbd resource:

drbdadm -- --overwrite-data-of-peer primary nfs

Example:

root@eave:~# drbdadm -- --overwrite-data-of-peer primary nfs
root@eave:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: 00000000000000000000000 
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:810112 nr:0 dw:0 dr:811672 al:0 bm:49 lo:0 pe:2 ua:7 ap:0 ep:1 wo:f oos:25404416
	[>....................] sync'ed:  3.2% (24808/25600)Mfinish: 0:40:52 speed: 10,340 (10,124) K/sec

How to change the state of drbd resource as secondary:

# drbdadm secondary nfs

How to change the state of drbd resource as primary:

# drbdadm primary nfs

How to change sync rate without stop/start drbd:

# drbdsetup /dev/drbd0 syncer -r 10M

You can do it also by modifying the config file and execute afterwards (assuming the resource is nfs):

# drbdadm adjust nfs