Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(LocalFeedRepository): speed up local feed extraction #7214

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

FineFindus
Copy link
Contributor

Speeds up the local feed extraction, by fetching more feeds concurrently and only applying a large timeout after fetching the full channels. This is based on the idea that YouTube's RSS feeds (extracted by the generic FeedInfo) have no/much less rate-limiting, likely due to the nature of RSS readers. I couldn't find an explicit statement from YouTube about that, but it makes sense. I've also increased to amount of concurrently fetched feeds in my fork for the last couple of months and had no noticeable impact so far.
A small timeout after each chunk is still applied, to be a good citizen of the web and not overwhelm/spam the server.

The used values may need to be adjusted in the future, either to allow for even faster extraction or throttling if a rate-limit is introduced.

Speeds up the local feed extraction, by fetching more feeds concurrently
and only applying a large timeout after fetching the full channels. This
is based on the idea that YouTube's RSS feeds (extracted by the generic
`FeedInfo`) have no/much less rate-limiting, likely due to the nature of
RSS readers.
A small timeout after each chunk is still applied, to be a good citizen
of the web and not overwhelm/spam the server.

Ref: TeamNewPipe/NewPipe#11817
@NeeRaj-2401
Copy link
Contributor

NeeRaj-2401 commented Mar 16, 2025

I was just thinking that instead of waiting until all the sub-channels are extracted before updating the feed, we could update it as soon as new content is extracted from a channel. This would make the user feel that the feed is being updated more quickly, enhancing the overall UX.

@FineFindus
Copy link
Contributor Author

FineFindus commented Mar 16, 2025

While I agree that immediately showing new items is a good idea, I doubt a good implementation is possible, as we don't know which channels have uploaded (and in which order), so the new entries would constantly be reshuffled, which I think would be rather confusing and possibly nauseating. Additionally, constantly adding new items at (essentially) random points in the recycler view might lead to performance problems.

That aside, I don't see how this is related to this PR? I think it would be better to open a new issue.

@FineFindus FineFindus marked this pull request as ready for review March 17, 2025 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants