Skip to content

/ glusterfs

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

glusterfsd of user 0 killed by SIGSEGV,myabe memory errors #1832

Open
perrynzhou opened this issue Nov 23, 2020 · 15 comments
Open

glusterfsd of user 0 killed by SIGSEGV,myabe memory errors #1832

perrynzhou opened this issue Nov 23, 2020 · 15 comments

Comments

@perrynzhou
Copy link
Contributor

@perrynzhou perrynzhou commented Nov 23, 2020

@xhernandez @amarts @BarakSason @mohit84 @tshacked @csabahenk @pranithk glusterfsd kill by sigsegv in our prod.we enable trash feature.i think that may be a bug

os version

# uname -r
3.10.0-693.el7.x86_64

gluster --version

# gluster --version
glusterfs 7.2

messages logs

Nov 22 15:27:07 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149502]: Process 16217 (glusterfsd) of user 0 killed by SIGSEGV - dumping core
Nov 22 15:27:10 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149502]: Failed to create core_backtrace: waitpid failed: No child processes
Nov 22 15:27:10 ai-storage-center-prd-10-194-8-2 abrt-server[149511]: Package 'glusterfs-fuse' isn't signed with proper key
Nov 22 15:27:10 ai-storage-center-prd-10-194-8-2 abrt-server[149511]: 'post-create' on '/var/spool/abrt/ccpp-2020-11-22-15:27:07-16217' exited with 1
Nov 22 15:27:10 ai-storage-center-prd-10-194-8-2 abrt-server[149511]: Deleting problem directory '/var/spool/abrt/ccpp-2020-11-22-15:27:07-16217'
Nov 22 15:27:10 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149514]: Process 16237 (glusterfsd) of user 0 killed by SIGSEGV - ignoring (repeated crash)
Nov 22 15:27:11 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149514]: Saved core dump of pid 16237 to core.16237 at / (557330432 bytes)
Nov 22 15:27:12 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149521]: Process 16257 (glusterfsd) of user 0 killed by SIGSEGV - ignoring (repeated crash)
Nov 22 15:27:14 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149521]: Saved core dump of pid 16257 to core.16257 at / (584548352 bytes)
@perrynzhou
Copy link
Contributor Author

@perrynzhou perrynzhou commented Nov 23, 2020 •
edited

messages logs-1

# gdb glusterfs /core.16237 
(gdb) bt
#0  0x00007fdeac39f173 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#1  0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#2  0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#3  0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#4  0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#5  0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#6  0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#7  0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#8  0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#9  0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#10 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#11 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#12 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
@perrynzhou
Copy link
Contributor Author

@perrynzhou perrynzhou commented Nov 23, 2020

messages logs-2

#13 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#14 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#15 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#16 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#17 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#18 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#19 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#20 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#21 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#22 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#23 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#24 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#25 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#26 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#27 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#28 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#29 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#30 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#31 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#32 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#33 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#34 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#35 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#36 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#37 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#38 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#39 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#40 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#41 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#42 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#43 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#44 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#45 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#46 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#47 0x00007fdea79dc076 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#48 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
@perrynzhou
Copy link
Contributor Author

@perrynzhou perrynzhou commented Nov 23, 2020

message logs-3

#49 0x00007fdea79dc4d8 in trash_unlink_mkdir_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#50 0x00007fdeac39f692 in posix_mkdir () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
---Type <return> to continue, or q <return> to quit---
#51 0x00007fdea79e323f in trash_unlink_rename_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#52 0x00007fdeac3a47f6 in posix_rename () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#53 0x00007fdea79e4513 in trash_unlink_stat_cbk () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#54 0x00007fdeac3aaa93 in posix_stat () from /usr/lib64/glusterfs/7.2/xlator/storage/posix.so
#55 0x00007fdea79e55a7 in trash_unlink () from /usr/lib64/glusterfs/7.2/xlator/features/trash.so
#56 0x00007fdea779daee in changelog_unlink () from /usr/lib64/glusterfs/7.2/xlator/features/changelog.so
#57 0x00007fdea735e482 in br_stub_unlink () from /usr/lib64/glusterfs/7.2/xlator/features/bitrot-stub.so
#58 0x00007fdea714a4e4 in posix_acl_unlink () from /usr/lib64/glusterfs/7.2/xlator/features/access-control.so
#59 0x00007fdea6f0f68e in pl_unlink () from /usr/lib64/glusterfs/7.2/xlator/features/locks.so
#60 0x00007fdea6cfc9f8 in worm_unlink () from /usr/lib64/glusterfs/7.2/xlator/features/worm.so
#61 0x00007fdea6aeb1f7 in ro_unlink () from /usr/lib64/glusterfs/7.2/xlator/features/read-only.so
#62 0x00007fdea68d8ce0 in leases_unlink () from /usr/lib64/glusterfs/7.2/xlator/features/leases.so
#63 0x00007fdea66c6b90 in up_unlink () from /usr/lib64/glusterfs/7.2/xlator/features/upcall.so
#64 0x00007fdebab7b683 in default_unlink_resume () from /lib64/libglusterfs.so.0
#65 0x00007fdebaaf01f5 in call_resume () from /lib64/libglusterfs.so.0
#66 0x00007fdea64a6e88 in iot_worker () from /usr/lib64/glusterfs/7.2/xlator/performance/io-threads.so
#67 0x00007fdeb990add5 in start_thread () from /lib64/libpthread.so.0
#68 0x00007fdeb91d2ead in clone () from /lib64/libc.so.6
(gdb)
@perrynzhou perrynzhou mentioned this issue Nov 25, 2020
Open
@pranithk
Copy link
Member

@pranithk pranithk commented Nov 25, 2020

@perrynzhou trash feature was never stable as per my understanding. It is disabled by default.
@pranithk
Copy link
Member

@pranithk pranithk commented Nov 25, 2020

@perrynzhou Do you have 'trash' feature enabled?
@perrynzhou
Copy link
Contributor Author

@perrynzhou perrynzhou commented Nov 25, 2020 •
edited

@perrynzhou Do you have 'trash' feature enabled?

@pranithk yes we enable trash ,Since it is unstable, why not remove it from glusterfs. At the same time, we don’t know which features are stable. There is no information to tell us which features are stable and which are unstable.we can't testing feature in our prod.at same time,glusterfs doc is keep last new.some doc description only valid on gluster 3.x such nfs-ganesha integration with glusterfs
@pranithk
Copy link
Member

@pranithk pranithk commented Nov 25, 2020

@amarts @xhernandez @thotz @anoopcs9 What do you guys think?
@pranithk
Copy link
Member

@pranithk pranithk commented Nov 25, 2020

@perrynzhou Do you have 'trash' feature enabled?

@pranithk yes we enable trash ,Since it is unstable, why not remove it from glusterfs. At the same time, we don’t know which features are stable. There is no information to tell us which features are stable and which are unstable.we can't testing feature in our prod.at same time,glusterfs doc is keep last new.some doc description only valid on gluster 3.x such nfs-ganesha integration with glusterfs

Features disabled are not stable. Don't enable them.
@pranithk
Copy link
Member

@pranithk pranithk commented Nov 25, 2020

@perrynzhou Do you have 'trash' feature enabled?

@pranithk yes we enable trash ,Since it is unstable, why not remove it from glusterfs. At the same time, we don’t know which features are stable. There is no information to tell us which features are stable and which are unstable.we can't testing feature in our prod.at same time,glusterfs doc is keep last new.some doc description only valid on gluster 3.x such nfs-ganesha integration with glusterfs

Features disabled are not stable. Don't enable them.

Actually you have a point, there are some features that are disabled by default but are stable. We will see what can be done about this.
@mykaul
Copy link
Contributor

@mykaul mykaul commented Nov 25, 2020

@perrynzhou Do you have 'trash' feature enabled?

@pranithk yes we enable trash ,Since it is unstable, why not remove it from glusterfs. At the same time, we don’t know which features are stable. There is no information to tell us which features are stable and which are unstable.we can't testing feature in our prod.at same time,glusterfs doc is keep last new.some doc description only valid on gluster 3.x such nfs-ganesha integration with glusterfs

Features disabled are not stable. Don't enable them.

In production.

But also, @perrynzhou , if you have such a large production setup (and I'm very glad to hear that), you should also have a staging environment for testing.
@mohit84
Copy link
Contributor

@mohit84 mohit84 commented Nov 25, 2020

@perrynzhou
Can you please share the output of the command "thread apply all bt full" after attach a core with gdb?
@perrynzhou
Copy link
Contributor Author

@perrynzhou perrynzhou commented Nov 26, 2020

@pranithk
Copy link
Member

@pranithk pranithk commented Nov 26, 2020

I informed the maintainers. A decision will be taken based on whether the goal will be to stabilize the feature or not.
@anoopcs9
Copy link

@anoopcs9 anoopcs9 commented Nov 26, 2020 •
edited

@xhernandez @amarts @BarakSason @mohit84 @tshacked @csabahenk @pranithk glusterfsd kill by sigsegv in our prod.we enable trash feature.i think that may be a bug

os version

# uname -r
3.10.0-693.el7.x86_64

gluster --version

# gluster --version
glusterfs 7.2

messages logs

Nov 22 15:27:07 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149502]: Process 16217 (glusterfsd) of user 0 killed by SIGSEGV - dumping core
Nov 22 15:27:10 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149502]: Failed to create core_backtrace: waitpid failed: No child processes
Nov 22 15:27:10 ai-storage-center-prd-10-194-8-2 abrt-server[149511]: Package 'glusterfs-fuse' isn't signed with proper key
Nov 22 15:27:10 ai-storage-center-prd-10-194-8-2 abrt-server[149511]: 'post-create' on '/var/spool/abrt/ccpp-2020-11-22-15:27:07-16217' exited with 1
Nov 22 15:27:10 ai-storage-center-prd-10-194-8-2 abrt-server[149511]: Deleting problem directory '/var/spool/abrt/ccpp-2020-11-22-15:27:07-16217'
Nov 22 15:27:10 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149514]: Process 16237 (glusterfsd) of user 0 killed by SIGSEGV - ignoring (repeated crash)
Nov 22 15:27:11 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149514]: Saved core dump of pid 16237 to core.16237 at / (557330432 bytes)
Nov 22 15:27:12 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149521]: Process 16257 (glusterfsd) of user 0 killed by SIGSEGV - ignoring (repeated crash)
Nov 22 15:27:14 ai-storage-center-prd-10-194-8-2 abrt-hook-ccpp[149521]: Saved core dump of pid 16257 to core.16257 at / (584548352 bytes)

Can you share gluster volume info <VOLNAME> output? What exactly was the operation performed?
@perrynzhou
Copy link
Contributor Author

@perrynzhou perrynzhou commented Nov 26, 2020

@anoopcs9 i already disable feature.trash.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
@pranithk @mykaul @anoopcs9 @perrynzhou @mohit84
You can’t perform that action at this time.