You can now use SQL to query the SEC in real time

Datamule now supports querying the SEC, such as form 345 tables as they update.

How this works:

SEC filing is detected.
Filing is processed, e.g. converted to tabular form.
Processed file is uploaded to AWS S3.
AWS Athena (SQL) can use the file.
At 06:00 UTC, file is combined with other files into a 'daily' dataset to reduce size due to better compression.

How you can use this:

default using datamulehub is to use up to date data.
however, due to how AWS Athena works, this query is more expensive. To save on costs use mode='daily', which scans just the daily dataset.

from datetime import datetime, timedelta
from zoneinfo import ZoneInfo

from datamulehub import databases


start = (
    datetime.now(ZoneInfo("America/New_York")).date()
    - timedelta(days=7)
).isoformat()

sql = f"""
SELECT
    t.accessionnumber,
    t.filingdate,
    t.issuertradingsymbol,
    TRY(date_parse(
        f.acceptance_datetime, '%Y%m%d%H%i%s'
    )) AS accepted_at
FROM "345" t
JOIN filing_metadata_filings f
    ON t.accessionnumber = f.accessionnumber
WHERE t.filingdate >= '{start}'
ORDER BY accepted_at DESC NULLS LAST, t.accessionnumber DESC
LIMIT 5
"""

for mode in ["daily", "both"]:
    rows = databases.read_query(
        sql,
        from_storage=mode,
    )
    print(mode, rows)