-
- Downloads
[+][FCB] Cats effects was giving warning on starvation, increase throughput...
[+][FCB] Cats effects was giving warning on starvation, increase throughput performance for delta messages via chunks + 1st it was attempted to make it fully granular, but...it does not behave well. Keep it in a branch and check another time. + It feels like every message on the redis stream opens a whole IO thing and the controller was eating all the memory very fast. + By grouping/chunking the problem gets put under certain control. + Testing showed to handle 20K delta items with grace, not instant, but gracefully. Felt like a good balance for redis and the controller. + Smaller chunks was giving starvation warnings, larger it's quite impactful on memory due to deser. + Maybe it's time to do binary streams, specially in this particular area of the code. + In any case, 500 chunks handles 20K properly. Still not the desired performance, but also, IPFS cluster is quite a bottleneck here. It's kind of a tricky problem. Newcomers, specially big members with over >100K files might find it unstable to put their shared folder online and this would be their very first experience with the software... But also...what are the chances that such a big member would it find unacceptable to have to chunk their file population? Still, no compromise, aim for handling millions of files with no problem. What's left then is: + Checking why does the CPU starvation happens. In >100K cases, the problem will still be there. + Binary streams. RPC to the rescue? This would need some serious metrics. + At some point, the file tree resolution will also act as a bottleneck. For 15K it's 2MBs in Redis and quite some memory pressure in controller's backend. Some solution will be needed here at some point.
Showing
- federation-controller-backend/src/main/scala/acab/devcon0/domain/adapters/federation/IpfsCidDeltaUpdateCommandHandlerImpl.scala 15 additions, 3 deletions...ers/federation/IpfsCidDeltaUpdateCommandHandlerImpl.scala
- federation-controller-backend/src/main/scala/acab/devcon0/domain/adapters/federationmember/SharingFolderUpdateCommandHandlerImpl.scala 2 additions, 3 deletions...erationmember/SharingFolderUpdateCommandHandlerImpl.scala
- federation-controller-backend/src/main/scala/acab/devcon0/domain/codecs/IpfsCodecs.scala 0 additions, 6 deletions...rc/main/scala/acab/devcon0/domain/codecs/IpfsCodecs.scala
- federation-controller-backend/src/main/scala/acab/devcon0/domain/service/ipfscid/FacadeServiceImpl.scala 1 addition, 1 deletion...ab/devcon0/domain/service/ipfscid/FacadeServiceImpl.scala
- federation-controller-backend/src/main/scala/acab/devcon0/input/redispubsub/FederationIpfsCidDeltaUpdateListener.scala 11 additions, 12 deletions...ut/redispubsub/FederationIpfsCidDeltaUpdateListener.scala
- federation-controller-backend/src/main/scala/acab/devcon0/input/redispubsub/FederationMemberChangelogUpdateEventListener.scala 40 additions, 9 deletions...pubsub/FederationMemberChangelogUpdateEventListener.scala
- federation-controller-backend/src/main/scala/acab/devcon0/input/redispubsub/FederationMemberIpfsCidDeltaUpdateListener.scala 34 additions, 13 deletions...ispubsub/FederationMemberIpfsCidDeltaUpdateListener.scala
- federation-controller-backend/src/main/scala/acab/devcon0/input/redispubsub/Listener.scala 1 addition, 1 deletion.../main/scala/acab/devcon0/input/redispubsub/Listener.scala
- federation-controller-backend/src/main/scala/acab/devcon0/output/publisher/RedisPublisherImpl.scala 14 additions, 0 deletions...la/acab/devcon0/output/publisher/RedisPublisherImpl.scala
- federation-controller-backend/src/main/scala/acab/devcon0/output/repository/redisutils/Redis.scala 4 additions, 1 deletion...ala/acab/devcon0/output/repository/redisutils/Redis.scala
Loading
Please register or sign in to comment