LimeWire 3.9.10

Matamoros · #41 (**permalink**) May 15th, 2004

Ok, I am now back-tracking from version 3.9.12 trying to discover in which version the CPU hogging starts.

At version 3.9.10, after ~60s , rather than the CPU jumping to 99%, I get a the following error message (twice) during a lot of disk activity. Is it something to do with ID3v2 parsing? In other words, in subsequent versions, does something get caught in a loop rather than generate an error???

LimeWire version 3.9.9 Pro
Java version 1.4.2_01 from Sun Microsystems Inc.
OS/2 v. 20.45 on x86
Free/total memory: 30632248/66650112

de.vdheide.mp3.ID3v2BadParsingException
at de.vdheide.mp3.ID3v2Frame.<init>(ID3v2Frame.java:2 31)
at de.vdheide.mp3.ID3v2.readFrames(ID3v2.java:936)
at de.vdheide.mp3.ID3v2.<init>(ID3v2.java:92)
at de.vdheide.mp3.ID3v2.<init>(ID3v2.java:118)
at com.limegroup.gnutella.mp3.ID3Reader.parseID3v2Dat a(ID3Reader.java:228)
at com.limegroup.gnutella.mp3.ID3Reader.parseFile(ID3 Reader.java:131)
at com.limegroup.gnutella.mp3.ID3Reader.readDocument( ID3Reader.java:115)
at com.limegroup.gnutella.xml.LimeXMLReplyCollection. constructDocument(LimeXMLReplyCollection.java:245)
at com.limegroup.gnutella.xml.LimeXMLReplyCollection. <init>(LimeXMLReplyCollection.java:198)
at com.limegroup.gnutella.xml.MetaFileManager.loadSet tingsBlocking(MetaFileManager.java:304)
at com.limegroup.gnutella.FileManager$1.managedRun(Fi leManager.java:592)
at com.limegroup.gnutella.util.ManagedThread.run(Mana gedThread.java:49)
Parent Cause:
java.lang.OutOfMemoryError

-- listing session information --
Current thread: FileManager.loadSettingsBlocking
Active Threads: 39
Uptime: 1:39
Is Connected: true
Number of Ultrapeer -> Ultrapeer Connections: 0
Number of Ultrapeer -> Leaf Connections: 0
Number of Leaf -> Ultrapeer Connections: 6
Number of Old Connections: 0
Acting as Ultrapeer: false
Acting as Shielded Leaf: true
Number of Active Uploads: 4
Number of Queued Uploads: 0
Number of Active Managed Downloads: 0
Number of Active HTTP Downloaders: 0
Number of Waiting Downloads: 0
Received incoming this session: true
Number of Shared Files: 4885
Guess Capable: true

-- listing threads --
Acceptor: 1
AWT-Shutdown: 1
AWT-EventQueue-0: 1
TreeHashTread: 1
pinger thread: 1
QRPPropagator: 1
MessageLoopingThread: 6
QueryDispatcher: 1
MulticastService: 1
HttpClient-ReferenceQueueThread: 1
TimerQueue: 1
FileManager.loadSettingsBlocking: 1
UDPService-Receiver: 1
TimerRunner: 1
OutputRunner: 6
ConnectionDispatchRunner: 4
QueryUnicaster: 1
ConnectionWatchdog: 1
Image Fetcher 0: 1
DestroyJavaVM: 1
Java2D Disposer: 1
PlayThread: 1
AWT-Windows: 1
HttpClient-IdleConnectionThread: 1
UDPService-Sender: 1
HTTPAcceptor: 1

-- listing properties --
LAST_EXPIRE_TIME=1084675839590
FILTER_ADULT=true
UPLOADS_PER_PERSON=1
SHOW_TOTD=false
CLIENT_ID=04D10FE24F6DF66BFF2E1BC688810700
FREELOADER_FILES=100
EVER_ACCEPTED_INCOMING=true
BANNED_WORDS=.wma;.rm;.m4a;.swf
FREELOADER_ALLOWED=10
MAX_UPLOAD_BYTES_PER_SEC=77
TOTAL_UPTIME=1700
PORT=6349
CONNECTION_VIEW_ENABLED=true
CLEAR_UPLOAD=false
AVERAGE_UPTIME=1700
EXTENSIONS_TO_SEARCH_FOR=mp3;ogg;jpg
INSTALLED=true
MAX_SIM_DOWNLOAD=8
LAST_GWEBCACHE_FETCH_TIME=1084677177130
CONNECTION_SPEED=350
MAX_DOWNLOAD_BYTES_PER_SEC=2

Matamoros · #42 (**permalink**) May 15th, 2004

First, thanks for the taking the time to address these issues; I realize that OS/2 is not a priority.

Quote:

Originally posted by verdyp
It seems to be an issue with the virtual memory management (with committed memory) on OS/2, and difficulties to handle multiple threads efficiently on this platform, and to support CPU throttling:

Perhaps. But I didn't have this problem until a recent beta.

Quote:

I note that you share a lot of files, but you have erased your THEX data cache. This is still a new feature in LimeWire, ant i will take time until all your files are indexed.

Oops. That is the filename of the THEX cache? Next time I will be more careful.

Quote:

However, throttling of this hasher by concurrent threads seems to be ineffective on OS/2.

Well,I am now running LimeWire 3.9.10 in the background (~14 uploads at the moment) and total CPU usage is around 5%. It is only versions 3.9.11 and 3.9.12 where I encounter this problem.

Quote:

I recommand you check your system settings related to virtual memory management (if you don't have a permanent and unfragmented swap space, you could experiment long delays o perform efficient multithreading).

I have 512 MB of memory, and the swap file (2MB) is unused, so fragmentation is not an issue.

Matamoros · #43 (**permalink**) May 15th, 2004

Starting with a fresh configuration several times in the course of various experiments this evening, I notice that LW (I am using v3.9.10 right now) apparently defaults to function as a leaf rather than a Ultra Peer, even though Disable UP is by default unchecked in the options.

To get around this, I checked this option, saved options, then unchecked it, save again, and then disconnected and reconnected. LW then (nearly instantly) connected as a UP.

Matamoros · #44 (**permalink**) May 15th, 2004

Quote:

Originally posted by verdyp

On Windows, my UltraPeer (using an Athlon XP 1800+) handles more than 450 concurrent threads with 5 to 10% of CPU usage...

At the moment, I have 3.9.10 running as UltraPeer with 12 uploads and 147 threads; in total my system has 340 active threads. CPU usage fluctates around 8-14%. I think OS/2 is also pretty good at multithreading.

verdyp · #45 (**permalink**) May 15th, 2004

Quote:

Originally posted by Matamoros
Starting with a fresh configuration several times in the course of various experiments this evening, I notice that LW (I am using v3.9.10 right now) apparently defaults to function as a leaf rather than a Ultra Peer, even though Disable UP is by default unchecked in the options.

To get around this, I checked this option, saved options, then unchecked it, save again, and then disconnected and reconnected. LW then (nearly instantly) connected as a UP.

I's normal that your servent starts as a leaf with a fresh configuraion: your public ip has no been validated, and there is no past proof of bandwidth and no past proof that your servent can accept incoming connections. Also, it needs to proove that your serven has a good past of connection availability.
The election mechanism needs several conditions before a node can become an UltraPeer. And even in that case, the promoion will only occur if you get replies from the network that more ultrapeers are needed. Of course this will only happen provided that Ultrapeer support is not disabled.
When you stop your servent, some statistics are computed and stored, and benefit to the next restart (the delay before resarting is taken into account to compute your host availability).

verdyp · #46 (**permalink**) May 15th, 2004

Quote:

Originally posted by Matamoros
At version 3.9.10, after ~60s , rather than the CPU jumping to 99%, I get a the following error message (twice) during a lot of disk activity. Is it something to do with ID3v2 parsing? In other words, in subsequent versions, does something get caught in a loop rather than generate an error???

We know that there are issues in ID3v2 parsing, because of he way it encodes some field sizes (and ambiguities in its past specifications).
If you have an example of such MP3 file that causes such issue, i would be good to upload it somewhere in a private URL, and to report this URL in a email to dev(at)core.limewire.org (don't post a file to this developers mailing list).
I had personnally a similar issue with a MP3 file that was created or tagged by an unknown codec or tagging tool. The solution o this problem was in a revision of the ID3v2 spec.
We have put some security for this parsing (because in the past it could cause LimeWire to not starting): now we track some malformed or unexpected formats, as well as the possible out of memory errors they could produce.
If your MP3 plays well in your player, or if your player displays the ID3v2 tag correctly, then we have another issue with an unsupported format. Now there is the ID3v2 spec, but there are also undocumented features used in some codecs or taggers, and sometimes too some bugs in various MP3 tagger tools.
The LimeWire's code tries to manage them by at least ignoring unparsable ID3v2 frames. If you know ID3 tags well, you'll see that no player today can read all possible formats (Windows Media Player for example is unable to parse many valid MP3 files and ignores many ID3 tag variants or more recent versions)

Matamoros · #47 (**permalink**) May 16th, 2004

Quote:

I's normal that your servent starts as a leaf with a fresh configuraion: your public ip has no been validated, and there is no past proof of bandwidth and no past proof that your servent can accept incoming connections. Also, it needs to proove that your serven has a good past of connection availability.

The election mechanism needs several conditions before a node can become an UltraPeer. And even in that case, the promoion will only occur if you get replies from the network that more ultrapeers are needed. Of course this will only happen provided that Ultrapeer support is not disabled.

When you stop your servent, some statistics are computed and stored, and benefit to the next restart (the delay before resarting is taken into account to compute your host availability).

Thanks for the very helpful explanation. In this and several earlier posts here, you've shared a lot of useful information about how LW works, and I hope that it can be added to the FAQ in some form.

With respect to that, I would be more than happy to volunteer my services: I am an Engish native-speaker who has worked as a (freelance) technical writer for many years. But of course I am not a programmer, and I can't tell how a program works just by looking at source code, so I would need a fair bit of input to be able to help develop the documentation further.

Anyway, let me know if I can be of assistance in any way; it is always said that programmers hate writing the documentation...

Matamoros · #48 (**permalink**) May 16th, 2004

Quote:

Originally posted by verdyp

If you have an example of such MP3 file that causes such issue, i would be good to upload it somewhere in a private URL, and to report this URL in a email to dev(at)core.limewire.org (don't post a file to this developers mailing list).

Yes, but you see the problem is that this is happening in the background (some 60 secs after Limewire v3.9.11/12 has scanned the Shared folders and established connections). I hear disk activity, it stops and CPU usages jumps up. Of course I have no idea which MP3 file is being read at the moment (if indeed that is what LW is doing). Perhaps you could add this to the information currently displayed very briefly at the bottom of the Search window on startup, but I can imagine you don't want the interface to get too busy.

In any case, I have backed down (temporarily I hope!) to 3.9.10 which doesn't have this problem.

verdyp · #49 (**permalink**) May 16th, 2004

Good idea: Limewire now integrates messages in the Status bar once the GUI construction is complete (they are displayed there instead of on the initial splash screen) and they explain what is being performed once the splash screen is closed and replaced by the constructed window.
There is now the support needed to display file names getting scanned (notably if they are MP3 with ID3v2 scan in progress). Note however that this status bar may become "flashy" if you add a large collection of files, such as when adding or setting a new shared folder.

stief · #50 (**permalink**) May 16th, 2004

[Thanks for the earlier info Philippe, and for all your gnutella/LW work too]

Well Sam--I couldn't break 3.9.12 on OS 10.3.3. I had to give after 24 hours running wide open because after 5GB up ands 1 GB down I used up my self-imposed bandwidth for the week! 4.0 is a great improvement and ready to roll as far as I can find--congrats to all once again.

Here are the only things that still seem odd:

--the 'already-downloaded icon' doesn't pick up on tunes that get sent to the shared iTunes folder if they are manually deleted from the unshared dl folder . (Roger--I don't work with iTunes much, so this is likely to be a new user problem. Otherwise, I was impressed by how well LW integrates with iTunes). I guess keeping the Library updated is in the future plans. The recognition of files in the unshared folder does work well--thanks for that.

--the incomplete icon doesn't always sort properly when complete and sorted by Quality (it will when the sort arrow is clicked twice). Too, some search results displayed stars in the Quality column but when selected for download, the 'overwrite?' message is properly displayed and the icon updates to the 'complete' one.

--several times the Monitor showed a duplicate upload (different progress) to the same IP/vendor. Uploading the same file at the same time seemed a waste of bandwidth, so I had to kill one (the lowest) and reset the upload prefs to only allow one slot per person. Too bad--I'd like to allow more slots per person.

--Ethernet transfers are still throttled by the upload bandwidth controls. Would be good if ethernet transfers were exempt.

---I did get one hang (beachball; Activity monitor registered a hang), but LW recovered within 20 seconds. This happened during the early minutes of a download swarming from 92 sources/downloading from 10 hosts. The file downloaded correctly.

So, after abusing LW by selecting >50 files at a time to dl or resume, changing upload prefs on the fly, quickly resuming multiple searches, messing with resuming incompletes from the library, and other nasty habits, I failed to break LW. I ran out of time (and bandwidth!) to try with OS 10.2.6 or Mac OS 8.6--sorry. I grabbed a bunch of stats screenshots before I had to quit

I hope the Opensourcer and Review pages will be updated in time for Tuesday. Too bad Jens and trap_jaw aren't around for a bit to also enjoy the reactions.

Thanks. Best wishes for the release.