How to get a historical Reddit RAG pack after Pushshift (without a 4TB torrent)

18 Aug 2026 · benthepythondev

Public Pushshift is gone. The official Reddit listing APIs still stop around a thousand posts, which on a busy subreddit is a few weeks. The leftover options in 2026 are a volunteer API that blinks, a multi-terabyte torrent, or a small structured pull you can actually embed.

I run the third one.

The job

A retrieval system needs what r/X said, not what is hot this morning. For most packs that means:

one or two subreddits
a date window (90 days is the default that fits a weekly refresh)
posts and comments
title / selftext / body / permalink / created_iso
a type field so you can split posts from comments

You do not need every comment Reddit ever stored. You need a window you can re-run.

What I do not recommend first

Academic Torrents and Arctic Shift monthly files are the right tool if you already have disk and a DuckDB habit. Watchful1’s dump is measured in terabytes. That is a warehouse project, not a RAG pack.

PullPush still speaks a Pushshift-shaped API. It also rate-limits and goes down. Fine behind a failover. Bad as the only pipe you sell to yourself.

Live Store scrapers are for monitoring. They see what a visitor sees today. They do not walk years of archive.

The pack I actually ship

Actor: Reddit Archive Scraper — Historical Posts for RAG

It pages PullPush and Arctic Shift, fails over when one archive is sick, and writes one row per post or comment. Public example: 90-day r/Python RAG pack.

Cloud proof already on that listing: run gJAFTYmczF0JICN6B, 119 items (25 posts + 94 comments) in about 38 seconds. That is a staging pack, not a backfill.

Store price stays $2.40 per 1,000 results on Gold. Use that if you will start the run. If you want 50k–1M rows delivered once, the packaged dump is on the Reddit RAG dump page: sample $250 (credited), then a fixed quote.

Dump prices Run the r/Python sample

A tight first input

{
  "subreddits": ["MachineLearning"],
  "afterDate": "2026-05-20",
  "maxPosts": 25,
  "includeComments": true,
  "maxCommentsPerPost": 20
}

Keep maxPosts small until the keyword and fields look right. Comments multiply the bill; leave them off if you only need thread titles and bodies.

What this is not

It is not a Pushshift clone. It is not user-history search. It is not comment-body full-text search across all of Reddit. Coverage follows the public archives. If a month is missing upstream, it is missing in the pack.

Use it for research and retrieval corpora. Do not use it to harass people.

Try it

Run the r/Python example Task.
If the fields fit, either schedule the Actor or send the subreddit + window from the dump page.
I will not email you first.

Affiliate if you still need an Apify account: apify.com/?fpr=qolupv. I may earn a commission; the price you pay does not change.