issues

Table actions

18 rows where comments = 1 and repo = 206156866 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: user, author_association, created_at (date), updated_at (date), closed_at (date)

type 2

state 2

repo 1

  • twitter-to-sqlite · 18 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
1077560091 I_kwDODEm0Qs5AOkMb 61 Data Pull fails for "Essential" level access to the Twitter API (for Documentation) jmnickerson05 57161638 open 0     1 2021-12-11T14:59:41Z 2022-10-31T14:47:58Z   NONE  

Per Twitter documentation: https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-leve

This isn't any fault of twitter-to-sqlite of course, but it should probably be documented as a side-note.

And this is how I'm surfacing the message from utils.py:

twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/61/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
1063982712 I_kwDODEm0Qs4_axZ4 60 Execution on Windows bernard01 1733616 open 0     1 2021-11-26T00:24:34Z 2022-10-14T16:58:27Z   NONE  

My installation on Windows using pip has been successful. I have Python 3.6.

How do I run twitter-to-sqlite? I cannot even figure out how "auth" is a command. I have python on my path: C:\prog\python\Python36;C:\prog\python\Python36\Scripts

Where should the commands be executed, and where are the files created?

Could some basics please be added to the documentation to get beginners started?

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/60/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
984939366 MDU6SXNzdWU5ODQ5MzkzNjY= 58 Error: Use either --since or --since_id, not both - still broken rubenv 42904 closed 0     1 2021-09-01T09:45:28Z 2021-09-21T17:37:41Z 2021-09-21T17:37:41Z CONTRIBUTOR  

Hi Simon,

It appears the fix for #57 doesn't fix things for me:

$ twitter-to-sqlite --version twitter-to-sqlite, version 0.21.4 $ python --version Python 3.9.6

$ twitter-to-sqlite home-timeline -a twitter-auth.json twitter/timeline.db --since Importing tweets Error: Use either --since or --since_id, not both

Is there any way I can help debug this?

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/58/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
984942782 MDExOlB1bGxSZXF1ZXN0NzI0MzE3NjUw 59 Fix for since_id bug, closes #58 rubenv 42904 closed 0     1 2021-09-01T09:49:09Z 2021-09-21T17:37:40Z 2021-09-21T17:37:40Z CONTRIBUTOR dogsheep/twitter-to-sqlite/pulls/59

Fixes remaining instances of this bug

 twitter-to-sqlite 206156866 pull     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/59/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
681575714 MDExOlB1bGxSZXF1ZXN0NDY5OTQ0OTk5 49 Document the use of --stop_after with favorites, refs #20 mikepqr 370930 closed 0     1 2020-08-19T06:10:52Z 2021-08-20T00:02:11Z 2021-08-20T00:02:11Z CONTRIBUTOR dogsheep/twitter-to-sqlite/pulls/49

(I discovered this trawling the issues for how to use --since with favorites)

 twitter-to-sqlite 206156866 pull     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
771324837 MDU6SXNzdWU3NzEzMjQ4Mzc= 53 --since support for favorites anotherjesse 27 closed 0     1 2020-12-19T07:08:23Z 2020-12-19T07:47:11Z 2020-12-19T07:47:11Z NONE  

Having support for --since for updating your favorites would be ideal as the api is both slow and it only returns ~3k most recent favorites.

https://twittercommunity.com/t/cant-get-all-favorite-tweets-by-rest-api/22007/3

The api seems to take an optional since_id parameter - https://developer.twitter.com/en/docs/twitter-api/v1/tweets/post-and-engage/api-reference/get-favorites-list

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
745393298 MDU6SXNzdWU3NDUzOTMyOTg= 52 Discussion: Adding support for fetching only fresh tweets fatihky 4169772 closed 0     1 2020-11-18T07:01:48Z 2020-11-18T07:12:45Z 2020-11-18T07:12:45Z NONE  

I think it'd be very useful if this tool has an option like --incremental to fetch only newer tweets. This way operations could complete very fast in sequential runs. I'd want to try to implement this feature if it seems OK for this tool's purpose.

twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
602619330 MDU6SXNzdWU2MDI2MTkzMzA= 45 Use raise_for_status() everywhere simonw 9599 open 0     1 2020-04-19T04:38:28Z 2020-04-19T04:39:22Z   MEMBER  

I keep seeing errors which I think are caused by authentication or rate limit problems but which appear to be unexpected JSON responses - presumably because they are actually an error message.

Recent example: https://github.com/simonw/jsk-fellows-on-twitter/runs/598892575

Using response.raise_for_status() everywhere will make these errors less confusing.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    
602176870 MDU6SXNzdWU2MDIxNzY4NzA= 43 "twitter-to-sqlite lists" command for retrieving a user's owned lists simonw 9599 closed 0     1 2020-04-17T19:08:59Z 2020-04-17T23:48:28Z 2020-04-17T23:30:39Z MEMBER   twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
585359363 MDU6SXNzdWU1ODUzNTkzNjM= 38 Screen name display for user-timeline is uneven simonw 9599 closed 0     1 2020-03-20T22:30:23Z 2020-03-20T22:37:17Z 2020-03-20T22:37:17Z MEMBER  

CDPHE [####################################] 67 CHFSKy [####################################] 3216 DHSWI [####################################] 41 DPHHSMT [####################################] 742 Delaware_DHSS [####################################] 3231 DhhsNevada [####################################] 639 I could format them to match the length of the longest screen name instead.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
561469252 MDExOlB1bGxSZXF1ZXN0MzcyMjczNjA4 33 Upgrade to sqlite-utils 2.2.1 simonw 9599 closed 0     1 2020-02-07T07:32:12Z 2020-03-20T19:21:42Z 2020-03-20T19:21:41Z MEMBER dogsheep/twitter-to-sqlite/pulls/33
twitter-to-sqlite 206156866 pull     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
515658861 MDU6SXNzdWU1MTU2NTg4NjE= 28 Add indexes to followers table simonw 9599 closed 0     1 2019-10-31T18:40:22Z 2019-11-09T20:15:42Z 2019-11-09T20:11:48Z MEMBER  

select follower_id from following where followed_id = 12497 takes over a second for me at the moment.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/28/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
513074501 MDU6SXNzdWU1MTMwNzQ1MDE= 26 Command for importing mentions timeline simonw 9599 closed 0     1 2019-10-28T03:14:27Z 2019-10-30T02:36:13Z 2019-10-30T02:20:47Z MEMBER   twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
506432572 MDU6SXNzdWU1MDY0MzI1NzI= 21 Fix & escapes in tweet text simonw 9599 closed 0     1 2019-10-14T03:37:28Z 2019-10-15T18:48:16Z 2019-10-15T18:48:16Z MEMBER  

Shouldn't be storing & here.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/21/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
503085013 MDU6SXNzdWU1MDMwODUwMTM= 13 statuses-lookup command simonw 9599 closed 0     1 2019-10-06T11:00:20Z 2019-10-07T00:33:49Z 2019-10-07T00:31:44Z MEMBER  

For bulk retrieving tweets by their ID.

https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-lookup

Rate limit is 900/15 minutes (1 call per second) but each call can pull up to 100 IDs, so we can pull 6,000 per minute.

Should support --SQL and --attach #8

twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
503045221 MDU6SXNzdWU1MDMwNDUyMjE= 11 Commands for recording real-time tweets from the streaming API simonw 9599 closed 0     1 2019-10-06T03:09:30Z 2019-10-06T04:54:17Z 2019-10-06T04:48:31Z MEMBER   twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
491791152 MDU6SXNzdWU0OTE3OTExNTI= 9 followers-ids and friends-ids subcommands simonw 9599 closed 0     1 2019-09-10T16:58:15Z 2019-09-10T17:36:55Z 2019-09-10T17:36:55Z MEMBER   twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/9/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
488874815 MDU6SXNzdWU0ODg4NzQ4MTU= 5 Write tests that simulate the Twitter API simonw 9599 open 0     1 2019-09-03T23:55:35Z 2019-09-03T23:56:28Z   MEMBER  

I can use betamax for this: https://pypi.org/project/betamax/

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);