Mitch Downey @mitch@podcastindex.social

@mitch @dave parsing seems stable by querying for updated feeds every minute. Average seems more like ~500 updated feeds per minute.

I’m surprised there are that many new episodes released online every minute. I thought I saw ~500 updated feeds every 10 minutes previously. I wonder if I messed up or something changed from the time I made that estimate?

Anyway, we seem stable now, but I definitely need to make our recent feeds firehose implementation more robust...

@dave it makes me wonder if anything has changed drastically recently? A week ago every 10 minutes was returning ~500 updated feeds for me. Now every 1 minute is returning ~700 updated feeds.

@dave uh oh, actually even if I change the interval to 1 minute since time, I am getting ~700 updated feeds in that time. I can't use 2 minutes since ~400 feeds will be missing. I'm going to change the interval to 1 minute, but it appears even that is too long to be reliable.

@mitch @dave I’m going to change it to 5 minutes since it’s missing feeds right now, but lmk if you have thoughts/concerns

@dave I've been asking the recently updated feeds endpoint for the max of 1000 feeds updated in the last 10 minutes, but it appears that time scale is not short enough to ensure less than 1000 feed results, so we're missing updates.

Can I reduce the interval that we are pinging that endpoint? How small can I make it without spamming?

@dave did I hear there is a podcast rankings endpoint? For sorting by popularity?

Database question:

I wanted to load our prod database on an external hard drive for local debugging purposes, and I did, but the process took over 24 hours to complete.

Basically I created the sql backup using pg_dump, then ran

psql -f /path/to/backup.sql

Does anyone know a faster way to restore a database from an sql file?

Since syncing our db with PI, search queries take way too long. They finish quick the second time you make a query (apparently bc the table is sorted in memory then) but the first request can take 20+ seconds.

My friend said our problem is the search query has a wildcard in the front, so the database can't use an index on the title column. The command:

'LOWER(podcast.title) LIKE :title':

title = '%Podcasting 2.0%'

I know it's a deep subject but anyone have advice for making search better?

The first podcast to use the <podcast:soundbite> tag with a title has been added to Podverse 🎉

We now have over 90000 soundbites in our database thanks to BuzzSprout, but soundbites only get featured on our home page if they have a title. The "Dudes And Dads Podcast" is the first podcast we know of to use the soundbite title attribute.

https://podverse.fm/clip/64h49iwo4j

@mitch @hypercatcher @martin @adam whoops meant to tag @dave 🤦‍♂️

An endpoint for retrieving crowdsourced clips is now live on Podverse! Provide the mp3 url for an episode to this endpoint and get a JSON containing all of its crowdsourced clip data.

For a demo check out the annotations for Podcasting 2.0 Episode 12:

https://api.podverse.fm/api/v1/clips?mediaUrl=https://mp3s.nashownotes.com/PC20-12-2020-11-20-Final.mp3

Questions and feedback appreciated!

@hypercatcher @martin @adam @martin

Podverse is now syncing with the Podcast Index 🎉 thanks to PI our database has grown from 90,000 podcasts to over 1.3 million! @adam @dave

Does anyone have theories as to what could be causing this? Signal 9? I'm using Docker and Postgres.

Dev question (Postgres):

I'm trying to load our fairly large prod Postgres database locally. I'm using an exported .sql file generated by pg_dump. In order to import the file I am using:

psql -W -f ../postgres.sql

However, I've run this 3 times, and after about 20 minutes, I get this crash:

LOG: background worker "parallel worker" (PID 94) was terminated by signal 9: Killed

Failed process was running: ALTER TABLE ONLY public.episodes
ADD CONSTRAINT "PK_blah" PRIMARY KEY (id)