"Loading Browse Data" failure on MyBook World Edit

[Bambleweeny57]

Hi all
{Note: re-posted from the General Set-up forum.}

Re: Soundbridge M1001 <-> mt_daapd Firefly on Western Digital MyBook World Edition II (MBWEII).

I recently installed the mt-daapd Firefly server on my MBWEII and I am experiencing problems browsing
my collection on my M1001 when the music collection served grows large (well,... annoyingly small actually but YNWIM!).

The connection between the mt-daapd server and the MBWEII and the M1001 seems fine.
Browsing SMALL collections on the MBWEII is also fine.
However, when I increase the number of songs in the database, at some point the Browse function begins to fail.
(initially just certain browse categories but eventually all as the collection grows)
Unfortunately, I can successfully serve only a very small fraction of my music collection from the MBWEII Firefly server before this problem kicks in bigtime!
My Full Firefly database on Win and MBWEII is approx 20MB, while my browsing functionality completely craps out with less than 25% of my collection loaded!


For reference,
- the M1001 is working fantastic on my Windows PC based Firefly server serving my entire (large) MP3 collection without fault.
- the music files served by both systems are identical.
- the Firefly database size on both servers are comparable.
- the Windows database seems to be sqlite3 while the MBWEII is not, it's a .gdb file.


As far as I can currently ascertain, there seems to be either;
a) a timeout on the browse function of the M1001 that is exceeded by the slower NSLU2 box transfer.
b) some limit on the number of items in the browse category transferred.
c) some specific tag handling problems that are only exposed as the collection grows AND that the two Firefly servers/databases handle differently.


Some Questions:

1, Is there any way to debug what exactly is failing?
I've enabled mt-daapd logging but it shows nothing when the browse fail occurs.
I need Roku Logging!

2. Is there any way to configure the Roku Browse limit/timeout? (if any exists)

3. Is there any way to make the Browse Info transfer more efficient from the mt-daapd server?

4. Is there any way to make the ipkg installer of mt-daapd configure the build with sqlite enabled?

5. should I actually be using the "svn" version of mt-daapd with this MBWEII?
(apart from this browse problem, the straight mt-daapd seems fine)



Any pointers would be very greatly appreciated,... Seriously, there could be hugs!
I'm dying to get my full collection on this NAS device.

Cheers
Paul

[andyg]

1, Is there any way to debug what exactly is failing?
I've enabled mt-daapd logging but it shows nothing when the browse fail occurs.
I need Roku Logging!

Turn up the debug to 9 through the mt-daapd config page and see if that throws any light.


2. Is there any way to configure the Roku Browse limit/timeout? (if any exists)

Not that I am aware of

3. Is there any way to make the Browse Info transfer more efficient from the mt-daapd server?

I guess you could use more playlists and filter your results that way but if you're browsing your entire library I don't think there's a way to make that any more efficient

4. Is there any way to make the ipkg installer of mt-daapd configure the build with sqlite enabled?

Pretty sure that's done automatically if you're using svn1586 or 1696

5. should I actually be using the "svn" version of mt-daapd with this MBWEII?
(apart from this browse problem, the straight mt-daapd seems fine)

Assuming you can install one of the Linux flavours available on the nightlies download page then yes. svn1586 or 1696 are the most reliable versions out there and were that last available before development stopped. You can get them at http://nightlies.fireflymediaserver.org/


If you're still having no joy then I'd suggest reposting on the Firefly forum - though it's currently down.

Andy

[Bambleweeny57]

Thanks so much for your input Andy.

I had debug at 7 but yes I'll try 9, that seems to be a frequent recommendation.


I have to admit that I just assumed the server wasn't using sqlite3 because;
- my Windows Firefly config file explicitly states db_type sqlite3 while the MBWEII config file does not
- the two database files have different extensions on the two platforms.
- the optware & ipkg mt_daapd install did not create the various sqlite package files in the filesystem.


I'll check my mt-daap version. I have to admit that I didn't pay too much attention as I installed opware,
did an ipkg update and then just installed mt-daapd.


I'll give the mt-daapd-svnXXXX a try instead of the plain mt-daapd.
Hopefully I'll avoid bricking anything!


I've done some more debug and it seems that the "bad tag" idea is not correct.
I incrementally added directories until I started getting browse failures.
Then re-built the library with just the latest directory and all browsing passed.
Basically, as the database gets bigger, the server begins to have problems extracting certain sub-categories of requested info.

The whole "timeout" thing also seems unlikely to, at least in an end-to-end sense.
I can have databases in which the "song" browse takes ages but passes while something much smaller like artists, fails.
I'm thinking that the mt-daapd server is just bogging down processing the browse list for some categories
(in a large database) and either sending corrupt data or failing a running timeout in communicating with the M1001.
I'll keep on keeping on plugging away until it makes sense!

Basically, as the M1001 works perfectly with my Windows Firefly serving my entire massive collection,
it's definitely a server side issue with the NSLU2 mt-daapd.
Most likely a horse-power issue with the platform when processing large songs.gbd files.

Thanks again for your kind input Andy

[bregent]

How many titles do you have in your entire library?

[andyg]

How many titles do you have in your entire library?

Bob's question is pertinent. Have a look at RokuMike's post here.

http://forums.roku.com/viewtopic.php?t=7505&highlight=load+size

[Bambleweeny57]

How many titles do you have in your entire library?

I'm afraid to say it's many tens of thousands, maybe 50K (not by my PC right now).
Now before y'all go reaching for the fire extinguisher for your hair,...

1. There are many many albums that are audio books or classical collections with a zillion tracks.
2. I never really want to access all these categories all at once,
but it'd be nice to have them in the one db so that, IF I feel like an audioboook or classical session one day,
I don't have to ssh into the device, swap db files and restart the server.
I really like the browse capability so it's a shame that it gets buggered by the db size,... ON THIS PLATFORM.
(As I said before, my entire 50k tracks or so browse perfectly and FAST between the Windows Firefly server and the M1001. So kudos to Roku!)

Anyways, I know, I'm a bad, bad man, but I want's what I want's.

Thanks to all for the pointers (and a modicom of forebearance)

Cheers
Paul

[Bambleweeny57]

I never really want to access all these categories all at once,
but it'd be nice to have them in the one db so that, IF I feel like an audioboook or classical session one day,

What would be really awesome for challenged platforms like this would be the ability to put multiple DB files in the DB directory
and then, when connecting to the server, be given a choice of what DB to load.
If you feel like some audio books, tell it to load that DB.
A bit of classical today? Hey presto bongo!
That'd be super fine!

[dupondt]

Hi Bambleweeny57,

I'm afraid to say it's many tens of thousands, maybe 50K

in this case your NAS presumably is the culprit. These boxes have very limited memory and thus can cause problems with large libraries (and yes, 50k tracks is a very large library). The linux experts who built a version of Firefly for my NAS box reckon that the limit might be around 10k tracks on my specific device. The same might be true for your WD box.

The solution in your case might be a Home Server on a Nettop PC or a Netbook.

Greetings
dupondt

[Bambleweeny57]

Hi Bambleweeny57,
The solution in your case might be a Home Server on a Nettop PC or a Netbook.

I suspect so.
In fact, a friend of mine did exactly that after meeting similar problems on another NSLU2 box!
I'll probably just whittle down my served library and just live with the NAS
(for when my main PC is off).
All-in-all it's pretty cool that it can even do what it does.

Thanks dupont.



P.S.
Actually I'm just a curious sort that's trying to work out WHY it fails more than anything else.
My best guess so far is that the mt-daapd server has issues when the compilation of the browse list gets more "complicated".
It often does just fine on even big lists like "Songs" while failing to extract the "Artist" or "Album" lists.
When I add more and more directories to the DB, especially "mixed" ones like Various Artist, I suspect that it has to do more complicated internal list management to generate browse request responses and at that point it fails.
Maybe there's a running time-out in the M1001 on comms from the server, maybe it just gets it's knickers in a twist and creates corrupt list.

[jjaf]

Just an idea, is it possible to run other instances of Firefly on the MBWEII? If you can break into the system, add another instance of FF to the services daemon with a different config file pointing to separate music and library folders.

[Bambleweeny57]

Just an idea, is it possible to run other instances of Firefly on the MBWEII? If you can break into the system, add another instance of FF to the services daemon with a different config file pointing to separate music and library folders.

Oooooh, you're a clever one,... you very nearly solved it there!


I added two firefly instances to my startup file, each using a unique *.conf file.
Each conf file had three unique fields;
- db_dir - location to put the songs.db file
- mp3_dir - pointer to the mp3 directory
- servername - Guess. No really, I have faith!

So I fired up the start script and,....
1. two sets of mt-daapd processes kicked off.
2. two unique songs.db files began accumulating, one in each db_dir location.
3. when the smaller server had finished IT WENT LIVE!
i.e. the M1001 could see it, files could be played,... and there was much rejoicing.
4. the larger server continued to process it's songs.db
5. when finished, the second server terminated. Oh the horror!

So close and yet so far.

I'll keep on investigating.
Perhaps there's some shared resource that I didn't configure differentially causing the second server to terminate.

Thanks for the great idea jjaf

[Bambleweeny57]

Cracked it!
Me being a dumbass, I forgot to give them unique port numbers.
Did that and it works great!

Thank you jjaf, you did it mate!


I'm really quite jazzed now!
Got this working AND also managed to complete a sweet dual 70mm fan replacement for the crappy, noisy little one in the MBWEII.
Now it runs super quiet!

I'd forgotten how great Unix is. Been in the Windows wilderness for several years.

Thanks to all for the input!

[jjaf]

I'm glad it worked for you, and pleasantly surprised that it turned out to be a straightforward task.

[Bambleweeny57]

Just in case anyone is interested in recreating this set-up, I thought I'd sign off this thread with a fuller description on the procedure
(in appreciation for the help this forum gave me)

Please Note:
For ANY of this to work, the MBWEII requires hacking to install the mt-daapd Firefly server (part of NSLU2 Optware package).
Easy to follow details can be found at the following links:
http://mybookworld.wikidot.com/hacks-and-howto
http://wiki.fireflymediaserver.org/MyBo ... on_Install
http://www.nslu2-linux.org/wiki/Optware/Mt-daapd


Once that is set up a small number of files need to be created and configured, as I shall attempt to detail below.

******************** FOR A SINGLE mt-daapd INSTANCE: ********************
1. Create the mt-daapd configuration file; /opt/etc/mt-daapd/mt-daapd.conf
Edit the the following fields:
-- db_dir - pointer to the directory holding the songs.gdb database file (e.g. /opt/var/mt-daapd)
-- mp3_dir - pointer to your mp3 directory (e.g /shared/internal/PUBLIC/MP3)
-- servername - a name for your server that will be visible to DAAP clients (e.g. MyBook Firefly)

2. create the songs database using the following command; /opt/sbin/mt-daapd -c /opt/etc/mt-daapd/mt-daapd.conf
This will start the mt-daapd process and begin creating/updating the songs.gdb database file.
You may monitor the progress of the database creation using the following commands
ls-l /opt/var/mt-daapd
ps -ef | grep mt-daapd

3. Create a script to start your mt-daap process with the MBWE (e.g. /opt/etc/init.d/S60mt-daapd)
Include the following:
#!/bin/sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/lib
if [ -n "`pidof mt-daapd`" ] ; then
killall mt-daapd 2>/dev/null
fi
sleep 2
/opt/sbin/mt-daapd -c /opt/etc/mt-daapd/mt-daapd.conf


******************** FOR MULTIPLE mt-daapd INSTANCES: ********************
For each mt-daapd instance (denoted with the generic "<xyz>" postfix below):

1. create a unique <xyz>.conf file in the directory /opt/etc/mt-daapd/.
Edit the following fields for each instance:
-- db_dir - location you wish to put the songs.gdb database file (e.g. /opt/var/mt-daapd/db_<xyz>/)
-- mp3_dir - pointer to the mp3 directory (e.g. /shared/internal/PUBLIC/MP3/<xyz>)
-- servername - (in MyBook Firefly <xyz>)
-- port - assign a unique port to each instance (e.g. 3689, 3690,...)
Optionally create unique entries for the following
-- rescan_interval - use different periods so that the db updates are usually not synchronized
-- playlist - pointer to customized playlist file for each songs.gdb (e.g. /opt/etc/mt-daapd/mt-daapd_<xyz>.playlist)
Note:
The mp3_dir field only accepts a single directory argument which means hacking your MP3 directory structure to suit your servers!
For greater flexibility, I personally point each "mp3_dir" field to a directory containing one (or more) symbolic links to the "branches"
of my MP3 library that I wish included in each database.
Creating one such "links" directory for each server, containing multiple links to branches of your MP3 tree is much easier.
For example, the following will include both the pop and blues subdirectories in the db<xyz>/songs.gdb database:
mkdir /opt/var/mt-daapd/db_<xyz>/ln
ln -s /shared/internal/PUBLIC/MP3/pop /opt/var/mt-daapd/db_<xyz>/ln/pop
ln -s /shared/internal/PUBLIC/MP3/blues /opt/var/mt-daapd/db_<xyz>/ln/blues

2. Create songs database with the following command:
/opt/sbin/mt-daapd -c /opt/etc/mt-daapd/mt-daapd_<xyz>.conf
This will start mt-daapd and begin creating/updating the database file indicated in the <xyz>.conf file
(/opt/var/mt-daapd/db_<xyz>/songs.gdb )

3. Create a script to start/restart mt-daapd with your MyBook (e.g. /opt/etc/init.d/S60mt-daapd)
#!/bin/sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/lib
if [ -n "`pidof mt-daapd`" ] ; then
killall mt-daapd 2>/dev/null
fi
sleep 2
# include the following line for each mtdaapd instance running:
/opt/sbin/mt-daapd -c /opt/etc/mt-daapd/mt-daapd_<xyz>.conf


That's about it really. Hopefully I have remembered all the necessary steps.
As long as your music collection doesn't exceed around 20,000 files, a single server works just fine.
I have three, serving over 40,000 songs. Works great. No sign of excessive processor loading.
-if that worries you, two of 70mm fans like below perfectly replace the noisy little bugger in the MBWEII
makes it nice and cool, plus even more noticeably, super quiet!
MASSCOOL FD07015S1M3/4 70mm Case Fan
http://www.newegg.com/Product/Product.a ... 6835150064