macphp asked:
I have a weird issue on an Ubuntu 10.04 LTS virtual server. A few days ago I started getting these alerts:
***** Nagios *****
Notification Type: PROBLEM
Service: Current Load
Host: localhost
Address: 127.0.0.1
State: CRITICAL
Date/Time: Mon Aug 13 13:09:58 EDT 2012
Now logging in to the server does not work. It seems like I am able to authenticate with ssh but then seems to fail at getting a prompt. I get:
/bin/bash: Exec format error
Connection to 192.172.132.20 closed.
I can’t login from any account. Can’t try single user as it is remotely hosted. I can connect without starting a shell using ssh -N.
I’ve received this from cron. It seems like all my /bin files are gone. Is there any way to recover?
/etc/cron.daily/apt:
/bin/sleep: 2: /bin/sleep: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/sleep: 1: /bin/sleep: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/sleep: 2: /bin/sleep: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É A8 uø‰ÈH^ZY[Ã: not found
/bin/sleep: 1: /bin/sleep: 1ÒŠ ˆ [email protected] ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/sleep: 1: /bin/sleep: ELF : not found
/bin/sleep: 1: /bin/sleep: Syntax error: word unexpected (expecting ")")
/bin/pidof: 2: /bin/pidof: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/pidof: 1: /bin/pidof: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/pidof: 2: /bin/pidof: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É A8 uø‰ÈH^ZY[Ã: not found
/bin/pidof: 1: /bin/pidof: 1ÒŠ ˆ [email protected] ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/pidof: 1: /bin/pidof: ELF : not found
/bin/pidof: 1: /bin/pidof: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/dpkg:
/bin/cp: 2: /bin/cp: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/cp: 1: /bin/cp: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/cp: 2: /bin/cp: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É A8 uø‰ÈH^ZY[Ã: not found
/bin/cp: 1: /bin/cp: 1ÒŠ ˆ [email protected] ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/cp: 1: /bin/cp: ELF : not found
/bin/cp: 1: /bin/cp: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/standard:
/bin/cp: 2: /bin/cp: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/cp: 1: /bin/cp: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/cp: 2: /bin/cp: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É A8 uø‰ÈH^ZY[Ã: not found
/bin/cp: 1: /bin/cp: 1ÒŠ ˆ [email protected] ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/cp: 1: /bin/cp: ELF : not found
/bin/cp: 1: /bin/cp: Syntax error: word unexpected (expecting ")")
/bin/awk: 2: /bin/awk: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/awk: 1: /bin/awk: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/awk: 2: /bin/awk: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É A8 uø‰ÈH^ZY[Ã: not found
/bin/awk: 1: /bin/awk: 1ÒŠ ˆ [email protected] ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/awk: 1: /bin/awk: ELF : not found
/bin/awk: 1: /bin/awk: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/thedailycommandline:
run-parts: failed to exec /etc/cron.daily/thedailycommandline: Exec format error
run-parts: /etc/cron.daily/thedailycommandline exited with return code 1
Any help would be greatly appreciated.
My answer:
That paste looks like severe filesystem corruption.
At this point, were it my server, I would:
- Check the disks for hardware failure and replace any disk that has failed, is failing, or even looks like it might fail in the near future.
- Reinstall the OS and restore my data from the most recent good backup, checking them carefully to ensure that I’m not restoring corrupt data. This could have been going on for quite a long time without your knowledge, so check your backup carefully before restoring it.
View the full question and any other answers on Server Fault.
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.