Posts by marmot
|
1)
Message boards :
Number crunching :
2 machines, thousands of WU's, 10 credit in 36 hours. What's wrong?
(Message 714)
Posted 22 Apr 2021 by marmot Post: All it takes is one corrupted VM in the Virtual Box manager and every one of these WU's is then Invalid. Need to kill the VBox service, start the management window and remove the corrupted entries, then restart the computer The restart is usually not needed but could not get valid WU's till after a restart. This means that any computer running these WU's will need a daily restart and VBox cleanup check. (This project has been the most annoying of all the ones I've done in last 5 years) Added a script killing any WU going longer than 3000 seconds. The error rate on those is over 90% and not worth the time wasting risk. |
|
2)
Message boards :
Number crunching :
Long work units run better with Linux (and so do short ones)
(Message 713)
Posted 22 Apr 2021 by marmot Post: I'm seeing massive numbers of invalids today that are completed by others. Not sure why this happens. Going to shut down VBox service and restart it to see if it's a management issue. EDIT 1: This might be the cause of a string of invalids. After VBox was restarted there was a missing media ('?' icon VM) NanoHub task in the VBox manager that had to be removed. Is it possible that the entire project has its slot indexes corrupted so it is attempting to interact with the wrong data\slot folder? If all the WU's still return invalid then I might have to manually delete the slot folders. EDIT 2: There are just 16 slots and the 1st 9 filled and the rest are empty, so they look fine. (One of the other machines getting all invalids had 450 data\slot folders filled with NanoHub WU remnants.) SAS RAID fragmentation minimal 0.45% which is also an improvement over the other machine with the 450 BOINC data\slots folders. It's fragmentation had risen to 15.5% |
|
3)
Message boards :
Number crunching :
Long work units run better with Linux (and so do short ones)
(Message 709)
Posted 21 Apr 2021 by marmot Post: Boinc2Docker doesn't work well with Virtual Box 6.x.x on Windows machines. I dropped back to VBox 5 on all machines that run Boinc2Docker. The machine that's actually running them most reliably is VBox 5.1.26 on Windows 8.1 with HDDTurbo RAM cache added (thanks Jim for the tip) I went through the valids and found only 6 over 5k seconds that completed in the last 2 days (but I have been aborting many that went over 59 mins): https://boinc.nanohub.org/nanoHUB_at_home/workunit.php?wuid=7960858 https://boinc.nanohub.org/nanoHUB_at_home/workunit.php?wuid=7937999 https://boinc.nanohub.org/nanoHUB_at_home/workunit.php?wuid=8021573 https://boinc.nanohub.org/nanoHUB_at_home/workunit.php?wuid=8049227 https://boinc.nanohub.org/nanoHUB_at_home/workunit.php?wuid=7919527 https://boinc.nanohub.org/nanoHUB_at_home/workunit.php?wuid=8047507 but I am nervous about letting them run past 1000 seconds as 79, (that I did not abort) over 3000 seconds in the last 2 days, got compute errors. The failure rate of the long runs are 79/85 = 93%. Too much wasted computing time. I'll allow 1 long run to complete in a batch of 8 WU's running and see if the failure rate improves. |
|
4)
Message boards :
Number crunching :
2 machines, thousands of WU's, 10 credit in 36 hours. What's wrong?
(Message 707)
Posted 20 Apr 2021 by marmot Post: All of my recent failed tasks were failed by at least one other person previously. That's why I said they are re-sends. And the project is out of work now so it's only sending back out tasks that have failed or that are aborted/reset. I do see that most of my tasks are, also, ending in Computation Error and are the 3rd to 5th attempt from prior sends. But my one machine, dedicated to only running these WU's, has completed 3 that were previously errors on other computers. (example: https://boinc.nanohub.org/nanoHUB_at_home/workunit.php?wuid=8021849). I think the RAM cache is crucial and I'm going to reduce my WU's from 10 at once to 8 and see if the reduced load improves the error rate. Are these WU's are extremely time dependent and must be done in under 30 minutes? (I'm now aborting any that have run over 30). At least, for the 1st time ever, I broke 2000 credit in a day. Yeah! |
|
5)
Message boards :
Number crunching :
2 machines, thousands of WU's, 10 credit in 36 hours. What's wrong?
(Message 704)
Posted 18 Apr 2021 by marmot Post: The project was actually running quite well for the first few days after new work became available, but it appears that all of the tasks that failed previously are being sent out again now, and almost everything is failing at this point since there aren't any more good tasks. Yours are showing Compute Errors; my bad WU's are marked Invalid. (You also have a 1060 3Gb/r9 280 configured machine). I've added RAM cache to all 3 servers but the two that were running WU's yesterday are only getting Invalid results, even though they appear to be running much more reliably (creating the entry in Virtual Box quickly, then entering computation quickly, and the drive thrashing is greatly reduced). The third machine, that just got WU's this afternoon, has 231 Valid, 0 Invalid and 3 Compute Errors. I'll reset the project on the 1st 2 machines and see if that has a positive result. |
|
6)
Message boards :
Number crunching :
Quite a few of my tasks end with 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED
(Message 702)
Posted 17 Apr 2021 by marmot Post:
That was a separate issue. Once I took Jim1348's advise about installing a RAM cache, the machine w/ VBox 5.1.26 is cranking through NanoHub WU's with no invalids. |
|
7)
Message boards :
Number crunching :
2 machines, thousands of WU's, 10 credit in 36 hours. What's wrong?
(Message 701)
Posted 17 Apr 2021 by marmot Post:
OK, so the performance increase is amazing. Thanks. The test machine is running 12 NanoHub WU in 32GB RAM. (That should be too many but they are peaking at different times and not actually using all their requested 3GB RAM in the VM settings). The RAM cache is peaking at about 5GB. My total credit is at 5700 as I post. See what it hits by Monday. Most the project WU's I'm trying to get 100k hours on are RAM hogs (Rosetta, YoYo ECM) but not disk intensive, so the RAM cache seemed optional in order to fit as many WU in RAM as possible. |
|
8)
Message boards :
Number crunching :
2 machines, thousands of WU's, 10 credit in 36 hours. What's wrong?
(Message 699)
Posted 17 Apr 2021 by marmot Post:
One server is shut down for warmer seasons approaching. I'll fire it back up today, try PrimoCache on that machine and it will try to do 8x NanoHub WU's and see how things go. Thanks. I'm going to stop all NanoHub work on the other 2 machines. ------- Oh, a correction Machine 2 (https://boinc.nanohub.org/nanoHUB_at_home/results.php?hostid=1687) was downgraded to VBox 5.2.44 so it could do Kryptos@Home. |
|
9)
Message boards :
Number crunching :
Quite a few of my tasks end with 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED
(Message 697)
Posted 17 Apr 2021 by marmot Post: Same here, a LOT of "EXIT_TIME_LIMITE_EXCEEDED" erros. From what I learned over at Kryptos@Home, which used Boinc2Docker as it's initial WU, Boinc2Docker does not work well (or at all) with Virtual Box 6.x. My guess is to run NanoHub you need to rollback to Virtual Box 5.2.44 (last update in version 5). Check VBox manager (after restarting computer or restarting the VBox service) and see if you have hundreds of invalid VM's built up. My 5.1.28 VBox machine is running these WU's (badly, with many invalids) and my 6.1.2 VBox machine has not completed any successful WU's even though it appears to be running them properly (after I limited the WU's to 4 max_concurrent in app_config.xml). I have some maintenance to perform on that machine and let you know if it succeeds to run a valid WU today. |
|
10)
Message boards :
Number crunching :
2 machines, thousands of WU's, 10 credit in 36 hours. What's wrong?
(Message 695)
Posted 17 Apr 2021 by marmot Post: Machine 1 (https://boinc.nanohub.org/nanoHUB_at_home/results.php?hostid=1675) stayed with Virtual Box 5.1.28 (because Boinc2Docker hasn't been updated in over a year - learned from Kryptos@Home earlier this year. See their forums.) and has run about 700 tasks. 39 valid, 630 invalid and about 30 errors. I know the machine ran out of RAM (even though BOINC auto limited to 8 WU's) but I manually reduced the WU's to 4 and the process manager reports that total commits never broke 31.4GB (RAM installed 32GB) so it would be barely into the swap. These WU's thrash the drives terribly upon copying a fresh VM to each new BOINC data slot folder. The invalids logs don't give a clue as to why they are invalid (unless I missed something; the errors all seem to be normal Boinc2Docker complaints about using a RAM drive). Machine 2 (https://boinc.nanohub.org/nanoHUB_at_home/results.php?hostid=1687) is updated to Virtual Box from March 2021 (6.1.2?) and wouldn't run Kryptos@Home Boinc2Docker WU's. It seems to be running Nano@home but... no valids shown and there are 445 slots in the data directory of BOINC and WUProps says it can't find any more drive storage even though the OS reports the drive has 143GB left. I think the naming index is full so doing maintenance after completely deleting all BOINC data slots (bye bye ATLAS WU's *sigh*). Something about VBoxManage ver 6.x.x with Boinc2Docker isn't removing the old WU's and building up project data slots under BOINC. Also, the VBox manager had hundreds of invalid VM's to be removed. Summary: I thought it'd be nice to get my 100k credit but the drive thrashing causes too much wear on the equipment, especially since the WU's are only a few minutes long. Only getting 40 valid WU's out of 660 for a total of 10 (TEN) credits in 18 hours of trying is just ridiculous return for my effort.. I'm going to finish maintenance then limit this project to 1 WU at a time and 0 resource share and see if things improve by Tuesday. |
|
11)
Message boards :
News :
Request for feedback
(Message 363)
Posted 27 Jun 2019 by marmot Post:
If you pause the WU's then open Virtual Box Manager and look at the settings on the boinc2docker; it has the VM's set to use 3072MB of RAM, although once running on the host, they never seem to require more than 1.7GB of committed (RAM+swap) on Windows 7 but haven't inspected every task and they could potentially peak at 3GB usage. |
|
12)
Message boards :
News :
Request for feedback
(Message 353)
Posted 13 Jun 2019 by marmot Post: @Nonexistent_Admin What are causing these errors: 10:26:44.089527 ERROR [COM]: aRC=VBOX_E_INVALID_OBJECT_STATE (0x80bb0007) aIID={b2547866-a0a1-4391-8b86-6952d82efaa0} aComponent={MachineWrap} aText={The given session is busy}, preserve=false aResultDetail=0 10:47:50.716886 ERROR [COM]: aRC=E_FAIL (0x80004005) aIID={b2547866-a0a1-4391-8b86-6952d82efaa0} aComponent={SessionMachine} aText={This machine does not have any snapshots}, preserve=false aResultDetail=0 10:47:51.325331 ERROR [COM]: aRC=VBOX_E_INVALID_OBJECT_STATE (0x80bb0007) aIID={4afe423b-43e0-e9d0-82e8-ceb307940dda} aComponent={MediumWrap} aText={Medium 'C:\Program Files\Oracle\VirtualBox\VBoxGuestAdditions.iso' is locked for reading by another task}, preserve=false aResultDetail=0 Also, noticed the Windows 8.1 machine with the invalids RAID 0 BOINC partition was underperforming by 35%. Reformated to 32kb blocks from 4kb blocks but still 100% invalid results. |
|
13)
Message boards :
News :
Request for feedback
(Message 342)
Posted 9 Jun 2019 by marmot Post: Analyzed your VM and saw it's OS resides on a 35MB CDROM image (impressive) and uses a shared folder to the hosts' BOINC slot directory for the app and data storage. All WU's appear to share the same OS and VBoxAdditions image. If I move to 8 of these machines at once, the failure rate over locked files will climb, perhaps to over 50%. Since the two images are under 100MB total, wouldn't it be easier to copy into each BOINC slot their own copy and eliminate the locked/missing image errors? |
|
14)
Message boards :
News :
Request for feedback
(Message 340)
Posted 8 Jun 2019 by marmot Post: The Windows 7 machine with similar hardware is running 100% valid results. Yes, then, but now it has a few errors of the same kind as the other machine with no snapshots and locked files. About 150 valid to 5 errors ~ 3%. It's running VBox 5.1.30. If you think it will fix the issue, I can try VBox 5.1.30 on the other two machines. My machines should be unhidden for you to see now. The HP laptop with Windows 7 running VBox 5.1.28 also and it ran 100% valid of the first nanoHUB BOINC2Dockers it tried while the Win 8.1 machine with 5.1.28 gets 100% invalids. |
|
15)
Message boards :
News :
Request for feedback
(Message 330)
Posted 7 Jun 2019 by marmot Post:
That machine w/ VBox ver 5.1.26 is now at 20 valid and 11 errors (used to get 100% invalids). 10:26:44.089527 ERROR [COM]: aRC=VBOX_E_INVALID_OBJECT_STATE (0x80bb0007) aIID={b2547866-a0a1-4391-8b86-6952d82efaa0} aComponent={MachineWrap} aText={The given session is busy}, preserve=false aResultDetail=0 The nearly identical machine in OS, updates, service configuration and hardware, w/ VBox ver 5.1.28 is getting 100% invalid results. Both machines running other projects, including Boinc2Docker VM's (BOINC@TACC, Cosmology@Home) successfully. The Windows 7 machine with similar hardware is running 100% valid results. |
|
16)
Message boards :
News :
Request for feedback
(Message 321)
Posted 6 Jun 2019 by marmot Post:
Decided to bring this project's personal credit to 10,000 points and see if things are better. Running on the same Windows 8.1 machine, with no changes to it's configuration, there were 6 completed, 0 invalids (last time near 100% invalids), 4 valid and 2 errors. Here's the last lines from a valid log: 2019-06-05 05:01:05 (4152): Guest Log: Running... Here's the error messages from the error WU. This is likely one of the 2 VM's that were suspended and restarted as I dropped the total WU count from 4 to 2 as the machine ran out of free RAM. 2019-06-05 05:01:05 (4920): Guest Log: Running... The machines don't take being suspended well? |
|
17)
Message boards :
News :
Request for feedback
(Message 289)
Posted 2 May 2019 by marmot Post:
Just wanted to add that the same Windows 8.1 machine with VBox 5.1.28 installed just successfully ran 227 valid boinc2docker work units received from BOINC@TACC. Not sure what is different between their version and yours that would make yours fail on identical hardware that runs theirs without an issue. EDIT: Wonder if that machine would run this boinc2docker if I reverted to VBox versions VBox versions Versions 5.1.12 – 5.1.18 or 5.1.22 - 5.1.26 |
|
18)
Message boards :
Number crunching :
Invalid rate.
(Message 279)
Posted 25 Apr 2019 by marmot Post: Actually open up the VM and interact with the long running WU and take a screen shot of it. The long runner I opened was sitting at a screen where, once I hit 'enter', it shut down and sent it's results. Wish I'd taken a screen shot; it was waiting on user input and I can't remember the phrasing of the prompt. |
|
19)
Message boards :
Number crunching :
Invalid rate.
(Message 272)
Posted 24 Apr 2019 by marmot Post: >>> Seems to be an issue with Windows 8.1 as the host OS. Those machines are running VM's all day, every day, and completed 20,000 hours of Theory VM's each, plus ran ATLAS and Cosmology camb_boinc2docker (beta test) which, I thought, was another version of the boinc2docker run here. The errors in the logs are the same whether valid or not (need to change the boot order to eliminate the earliest ones). Got any idea what to look for? Those 2 separate machines run everything else they've come across, so this is about finding out what is broken with these WU's. |
|
20)
Message boards :
News :
Request for feedback
(Message 269)
Posted 24 Apr 2019 by marmot Post:
My Windows 7 Professional 64 Xeon E5-2660 machine with VBox 5.1.30 finished 397 valid, 0 invalid and 1 error (196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED). Runs great on that machine, 100% invalid rate on the nearly identical hardware machines but with Windows 8.1. I'll try out some more on the next version. G'luck with the alpha testing. |
Next 20
©2024 COPYRIGHT 2017-2018 NCN