Commit Graph

73 Commits (8004e6f31cb03abc9b6170622099879ccaf5570b)

Author SHA1 Message Date
Justin Clark-Casey (justincc) 8004e6f31c Fix issue just introduced in 8efe4bfc2e where I accidentally left in a test line to force very quick client unack 2013-07-29 23:38:54 +01:00
Justin Clark-Casey (justincc) 8efe4bfc2e Make "abnormal thread terminations" into "ClientLogoutsDueToNoReceives" and add this to the StatsManager
This reflects the actual use of this stat - it hasn't recorded general exceptions for some time.
Make the sim extra stats collector draw the data from the stats manager rather than maintaing this data itself.
2013-07-29 23:18:29 +01:00
Diva Canto cac37e298c Deleted all [ZZZ] debug messages. 2013-07-24 14:31:30 -07:00
Diva Canto e6a0f6e428 One more thing to test in order to let CompleteMovement go up the stack. 2013-07-24 14:29:51 -07:00
Diva Canto 14530b2607 Minor adjustment on timings of waits. 2013-07-24 14:29:37 -07:00
Diva Canto 3891a8946b New Teleport protocol (V2), still compatible with V1 and older. (version of the destination is being checked)
In this new protocol, and as committed before, the viewer is not sent EnableSimulator/EstablishChildCommunication for the destination. Instead, it is sent TeleportFinish directly. TeleportFinish, in turn, makes the viewer send a UserCircuitCode packet followed by CompleteMovementIntoRegion packet. These 2 packets tend to occur one after the other almost immediately to the point that when CMIR arrives the client is not even connected yet and that packet is ignored (there might have been some race conditions here before); then the viewer sends CMIR again within 5-8 secs. But the delay between them may be higher in busier regions, which may lead to race conditions.
This commit improves the process so there are are no race conditions at the destination. CompleteMovement (triggered by the viewer) waits until Update has been sent from the origin. Update, in turn, waits until there is a *root* scene presence -- so making sure CompleteMovement has run MakeRoot. In other words, there are two threadlets at the destination, one from the viewer and one from the origin region, waiting for each other to do the right thing. That makes it safe to close the agent at the origin upon return of the Update call without having to wait for callback, because we are absolutely sure that the viewer knows it is in th new region.
Note also that in the V1 protocol, the destination was getting UseCircuitCode from the viewer twice -- once on EstablishAgentCommunication and then again on TeleportFinish. The second UCC was being ignored, but it shows how we were not following the expected steps...
2013-07-24 14:27:58 -07:00
Justin Clark-Casey (justincc) a57a472ab8 Add proper method doc and comments to m_dataPresentEvent (from d9d9959) 2013-07-23 00:51:59 +01:00
Justin Clark-Casey (justincc) 9fb9da1b6c Add clientstack.InboxPacketsCount stat. This records the number of packets waiting to be processed at the second stage (after initial UDP processing)
If this consistently increases then this is a problem since it means the simulator is receiving more requests than it can distribute to other parts of the code.
2013-07-23 00:35:41 +01:00
Justin Clark-Casey (justincc) 60732c96ef Add clientstack.OutgoingUDPSendsCount stat to show number of outbound UDP packets sent by a region per second 2013-07-23 00:35:34 +01:00
Justin Clark-Casey (justincc) 8396f1bd42 Record raw number of UDP receives as clientstack.IncomingUDPReceivesCount 2013-07-23 00:35:23 +01:00
Justin Clark-Casey (justincc) bf517899a7 Add AverageUDPProcessTime stat to try and get a handle on how long we're taking on the initial processing of a UDP packet.
If we're not receiving packets with multiple threads (m_asyncPacketHandling) then this is critical since it will limit the number of incoming UDP requests that the region can handle and affects packet loss.
If m_asyncPacketHandling then this is less critical though a long process will increase the scope for threads to race.
This is an experimental stat which may be changed.
2013-07-23 00:35:09 +01:00
Robert Adams e6b6af62dd Added check for user movement specification before discarding an incoming
AgentUpdate packet. This fixes the problem with vehicles not moving forward
after the first up-arrow.
Code to fix a potential exception when using different IClientAPIs.
2013-07-22 15:41:14 -07:00
Diva Canto 174105ad02 Fixed the stats in show client stats. Also left some comments with observations about AgentUpdates. 2013-07-21 09:00:27 -07:00
Justin Clark-Casey (justincc) 61eda1f441 Make the check as to whether any particular inbound AgentUpdate packet is significant much earlier in UDP processing (i.e. before we pointlessly place such packets on internal queues, etc.)
Appears to have some impact on cpu but needs testing.
2013-07-21 08:58:55 -07:00
Justin Clark-Casey (justincc) 5a2d4d888c Hack in console command "debug lludp toggle agentupdate" to allow AgentUpdate in packets to be discarded at a very early stage.
Enabling this will stop anybody from moving on a sim, though all other updates should be unaffected.
Appears to make some cpu difference on very basic testing with a static standing avatar (though not all that much).
Need to see the results with much higher av numbers.
2013-07-21 08:58:21 -07:00
Justin Clark-Casey (justincc) 3a476bf60c Fix up a temporary debugging change from last commit which stopped "lludp stop out" from actually doing anything 2013-07-21 08:57:36 -07:00
Justin Clark-Casey (justincc) 63c42d6602 Do some simple queue empty checks in the main outgoing udp loop instead of always performing these on a separate fired thread.
This appears to improve cpu usage since launching a new thread is more expensive than performing a small amount of inline logic.
However, needs testing at scale.
2013-07-21 08:56:48 -07:00
Justin Clark-Casey (justincc) d9d995914c try Hacking in an AutoResetEvent to control the outgoing UDP loop instead of a continuous loop with sleeps.
Does appear to have a cpu impact but may need further tweaking
2013-07-18 12:28:02 -07:00
Diva Canto 864f15ce4d Revert the revert
Revert "Trying to hunt the CPU spikes recently experienced."

This reverts commit ac73e70293.
2013-07-15 11:52:26 -07:00
Diva Canto ac73e70293 Trying to hunt the CPU spikes recently experienced.
Revert "Comment out old inbound UDP throttling hack. This would cause the UDP"

This reverts commit 38e6da5522.
2013-07-15 11:27:49 -07:00
Diva Canto a412b1d682 Moved SendInitialDataToMe to earlier in CompleteMovement. Moved TriggerOnMakeRootAgent to the end of CompleteMovement.
Justin, if you read this, there's a long story here. Some time ago you placed SendInitialDataToMe at the very beginning of client creation (in LLUDPServer). That is problematic, as we discovered relatively recently: on TPs, as soon as the client starts getting data from child agents, it starts requesting resources back *from the simulator where its root agent is*. We found this to be the problem behind meshes missing on HG TPs (because the viewer was requesting the meshes of the receiving sim from the departing grid). But this affects much more than meshes and HG TPs. It may also explain cloud avatars after a local TP: baked textures are only stored in the simulator, so if a child agent receives a UUID of a baked texture in the destination sim and requests that texture from the departing sim where the root agent is, it will fail to get that texture.
Bottom line: we need to delay sending the new simulator data to the viewer until we are absolutely sure that the viewer knows that its main agent is in a new sim. Hence, moving it to CompleteMovement.
Now I am trying to tune the initial rez delay that we all experience in the CC. I think that when I fixed the issue described above, I may have moved SendInitialDataToMe to much later than it should be, so now I'm moving to earlier in CompleteMovement.
2013-07-13 09:46:58 -07:00
Robert Adams 38e6da5522 Comment out old inbound UDP throttling hack. This would cause the UDP
reception thread to sleep for 30ms if the number of available user worker
threads got low. It doesn't look like any of the UDP packet types are
marked async so this check is 1) unnecessary and 2) really crazy since
it stops up the reception thread under heavy load without any indication.
2013-07-09 18:34:24 -07:00
Justin Clark-Casey (justincc) 16f40c1a15 Add --default option to "debug lludp packet" command to allow packet logging to be performed immediately from client start 2013-07-04 17:29:53 +01:00
Justin Clark-Casey (justincc) 25889b2d7e change "debug packet" command to "debug lludp packet" to conform with other "debug lludp" options
also moves the implementing code into LLUDPServer.cs along with other debug commands from OpenSim.cs
gets all debug lludp commands to only activate for the set scene if not root
2013-07-04 00:02:53 +01:00
Diva Canto 91091c3e54 Second take at HGTP-mesh bug: delay sending the initial data only for agents that are coming via TP (root agents) 2013-05-14 09:06:58 -07:00
Diva Canto 645da54f25 Revert "HGTP-mesh bug: the mesh download requests were going to the departing sims for a little while. This was also true for local TPs. BUt for local TPs the assets are on the same server, so it doesn't matter. For HGTPs, it matters. This potential fix moves sending the initial data to later, after the client has completed the movement into the region. Fingers crossed that it doesn't mess other things up!"
This reverts commit f32a21d967.
2013-05-14 08:47:18 -07:00
Diva Canto f32a21d967 HGTP-mesh bug: the mesh download requests were going to the departing sims for a little while. This was also true for local TPs. BUt for local TPs the assets are on the same server, so it doesn't matter. For HGTPs, it matters. This potential fix moves sending the initial data to later, after the client has completed the movement into the region. Fingers crossed that it doesn't mess other things up! 2013-05-14 08:12:01 -07:00
Justin Clark-Casey (justincc) c5de9840b0 refactor: Remove IClientNetworkServer.NetworkStop() in favour of existing Stop().
This was an undocumented interface which I think was for long defunct region load balancing experiments.
Also adds method doc for some IClientNetworkServer methods.
2013-04-16 21:58:24 +01:00
Robert Adams c10c43d6f6 Fix server statistics always reporting zero for total network bytes in/out.
Clean up some parameter code in Statistics.Binary.
2013-03-15 13:59:34 -07:00
Justin Clark-Casey (justincc) cd088757e9 Add a first draft mechanism for the IncomingPacketsProcessedStat to show the delta over time.
The chief motivation for this is to be able to tell whether there's any impact on incoming packet processing from enabling extra packet pooling.
2012-11-16 04:36:22 +00:00
Justin Clark-Casey (justincc) 038528dc80 Make PacketPool class stats pull stats instead of push stats so they can be lifted up into LLUDPServer and be distiguished by scene name 2012-11-15 02:02:59 +00:00
Justin Clark-Casey (justincc) 2c36106675 Add IncomingPacketsProcessedCount stat for diagnostics.
Also puts some packet processing counts in a container named after the scene so that stats can be collected from more than one scene.
2012-11-15 01:14:18 +00:00
Justin Clark-Casey (justincc) 319ebaca06 Make it possible to turn the base UDP object packet pools on and off whilst running via the "debug lludp pool <on|off>" console command. For debug purposes.
This does not currently apply to the higher LLUDP packetpool.
2012-10-23 02:44:15 +01:00
Justin Clark-Casey (justincc) 4578ff74fe Add object count stats for new IncomingPacket and UDPPacketBuffer pools if they are enabled. Add count stats for existing LLUDP pool.
This introduces a pull stat type in addition to the push stat type.
A pull stat takes a method on construction which knows how to update the stat on request.
In this way, special interfaces for pull stat collection are not necessary.
2012-10-23 01:52:10 +01:00
Justin Clark-Casey (justincc) faf6b56839 Explicitly return only the incoming AgentUpdate packet as this is the only one we pool atm, rather than attempting to return all incoming packets. 2012-10-17 23:08:14 +01:00
Justin Clark-Casey (justincc) 2ed59ad8ac If RecycleBaseUDPPackets = true, also pool IncomingPackets to reduce memory churn 2012-10-17 21:08:15 +01:00
Justin Clark-Casey (justincc) fc861c7904 Add optional pool for the UDPPacketBuffer objects that handle all incoming UDP data.
Even when an avatar is standing still, it's sending in a constant stream of AgentUpdate packets that the client creates new UDPPacketBuffer objects to handle.
This option pools those objects.  This reduces memory churn.
Currently off by default.  Works but the scope can be expanded.
2012-10-16 23:35:05 +01:00
Justin Clark-Casey (justincc) db4ca57590 Make it possible to separate start and stop lludp packet processing from the console for debug processes.
This is controlled via the "debug lludp start <in|out|all>" and "debug lludp stop <in|out|all>" region console commands.
The command "debug lludp status" will show current status.
2012-10-16 21:55:00 +01:00
Justin Clark-Casey (justincc) 21d0cbf703 Add AgentUpdate to PacketPool. This is the most common inbound packet from viewers. 2012-10-12 01:39:37 +01:00
Justin Clark-Casey (justincc) 2e9ef015f7 Fix packetpool for ImprovedTerseObjectUpdate packets.
These were neither being returned or in many places reused.
Getting packets from a pool rather than deallocating and reallocating reduces memory churn which in turn reduces garbage collection time and frequency.
2012-10-11 23:58:37 +01:00
Justin Clark-Casey (justincc) e76b01a201 Lock on AgentCircuitData during Scene.AddClient() and RemoveClient() to prevent an inactive connection being left behind if the user closes the viewer whilst the connection is being established.
This should remove the need to run the console command "kick user --force" when these connections are left around.
2012-10-10 00:26:43 +01:00
Justin Clark-Casey (justincc) 7c7cdf3431 Read PacketPool config in LLUDPServer with other config params rather than in Scene.
This is to resolve previous build break.
This unnecessarily but harmlessly reads and sets the parameter multiple times - scene was doing the same thing.
2012-10-05 01:43:29 +01:00
Justin Clark-Casey (justincc) 35efa88c26 Rename OpenSim.Framework.Statistics to OpenSim.Framework.Monitoring.
This better reflects the long-term purpose of that project and matches Monitoring modules.
2012-07-25 23:11:50 +01:00
Justin Clark-Casey (justincc) ba80f137b5 Prevent race conditions between two threads that call LLClientView.Close() simultaneously (e.g. ack timeout and an attempt to reconnect) 2012-07-19 22:32:27 +01:00
Justin Clark-Casey (justincc) d6f54b25cd Stop redundantly passing in the endpoint to the LLClientView constructor.
This can always be retrieved via the LLUDPClient and is so done in various places already.
2012-07-12 23:48:42 +01:00
Justin Clark-Casey (justincc) 5691a8b860 refactor: rename Watchdog.WATCHDOG_TIMEOUT_MS to DEFAULT_WATCHDOG_TIMEOUT_MS to reflect what it actually is 2012-07-04 00:15:03 +01:00
Justin Clark-Casey (justincc) aaa30dcebc Add region name to UseCircuitCode log messages 2012-06-15 05:01:36 +01:00
Justin Clark-Casey (justincc) 2ca31a9841 Remove accidental timeout left in during earlier debugging. Has been in since two commits ago (b099f26) 2012-06-12 02:46:14 +01:00
Justin Clark-Casey (justincc) b099f26376 Set IClientAPI.IsActive = false early on client removal due to ack timeout rather than using IsLoggingOut flag.
IsActive is more appropriate since unack timeout is not due to voluntary logout.
This is in line with operations such as manual kick that do not set the IsLoggingOut flag.
It's also slightly better race-wise since it reduces the chance of this operation clashing with another reason for client deactivation (e.g. manual kick).
2012-06-12 02:16:36 +01:00
Justin Clark-Casey (justincc) c89db34fc4 If the simulator closes a root agent due to ack timeout, then send the client a kick message with that reason, in case it is somehow still listening. 2012-06-12 02:03:31 +01:00