id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,pull_request,body,repo,type,active_lock_reason,performed_via_github_app,reactions,draft,state_reason
639542974,MDU6SXNzdWU2Mzk1NDI5NzQ=,47,Fall back to FTS4 if FTS5 is not available,73579,open,0,,,3,2020-06-16T10:11:23Z,2020-06-17T20:13:48Z,,NONE,,"got this with version 0.21.1 from pypi. twitter-to-sqlite auth worked but then ""twitter-to-sqlite user-timeline USER.db"" produced a tracekback ending in ""no such module: FTS5"". ",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
745393298,MDU6SXNzdWU3NDUzOTMyOTg=,52,Discussion: Adding support for fetching only fresh tweets,4169772,closed,0,,,1,2020-11-18T07:01:48Z,2020-11-18T07:12:45Z,2020-11-18T07:12:45Z,NONE,,I think it'd be very useful if this tool has an option like `--incremental` to fetch only newer tweets. This way operations could complete very fast in sequential runs. I'd want to try to implement this feature if it seems OK for this tool's purpose. ,206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
771324837,MDU6SXNzdWU3NzEzMjQ4Mzc=,53,--since support for favorites,27,closed,0,,,1,2020-12-19T07:08:23Z,2020-12-19T07:47:11Z,2020-12-19T07:47:11Z,NONE,,"Having support for `--since` for updating your favorites would be ideal as the api is both slow and it only returns ~3k most recent favorites.
https://twittercommunity.com/t/cant-get-all-favorite-tweets-by-rest-api/22007/3
The api seems to take an optional `since_id` parameter - https://developer.twitter.com/en/docs/twitter-api/v1/tweets/post-and-engage/api-reference/get-favorites-list",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
796736607,MDU6SXNzdWU3OTY3MzY2MDc=,56,Not all quoted statuses get fetched?,42315895,closed,0,,,3,2021-01-29T09:48:44Z,2021-02-03T10:36:36Z,2021-02-03T10:36:36Z,NONE,,"
In my database I have 13300 quote tweets, but eta 3600 have `quoted_status` empty.
I fetched some of them using `https://api.twitter.com/1.1/statuses/show.json?id=xx` and they did have ids of quoted tweets.",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1063982712,I_kwDODEm0Qs4_axZ4,60,Execution on Windows,1733616,open,0,,,1,2021-11-26T00:24:34Z,2022-10-14T16:58:27Z,,NONE,,"My installation on Windows using pip has been successful. I have Python 3.6.
How do I run twitter-to-sqlite? I cannot even figure out how ""auth"" is a command. I have python on my path:
C:\prog\python\Python36;C:\prog\python\Python36\Scripts
Where should the commands be executed, and where are the files created?
Could some basics please be added to the documentation to get beginners started?",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/60/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1077560091,I_kwDODEm0Qs5AOkMb,61,"Data Pull fails for ""Essential"" level access to the Twitter API (for Documentation)",57161638,open,0,,,1,2021-12-11T14:59:41Z,2022-10-31T14:47:58Z,,NONE,,"Per Twitter documentation:
https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-leve
This isn't any fault of twitter-to-sqlite of course, but it should probably be documented as a side-note.
And this is how I'm surfacing the message from utils.py:
",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/61/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1088816961,I_kwDODEm0Qs5A5gdB,62,KeyError: 'created_at' for private accounts?,6764957,closed,0,,,2,2021-12-26T17:51:51Z,2022-03-12T02:36:32Z,2022-02-24T18:10:18Z,NONE,,"hey Simon!
i was running `twitter-to-sqlite user-timeline twitter.db` for [my private alt](https://twitter.com/swyxio) and ran into this error:
```bash
Traceback (most recent call last):
File ""/Users/swyx/Work/datasette/env/bin/twitter-to-sqlite"", line 8, in
sys.exit(cli())
File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py"", line 1128, in __call__
return self.main(*args, **kwargs)
File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py"", line 1053, in main
rv = self.invoke(ctx)
File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py"", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py"", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py"", line 754, in invoke
return __callback(*args, **kwargs)
File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/cli.py"", line 291, in user_timeline
profile = utils.get_profile(db, session, **kwargs)
File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py"", line 133, in get_profile
save_users(db, [profile])
File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py"", line 453, in save_users
transform_user(user)
File ""/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py"", line 285, in transform_user
user[""created_at""] = parser.parse(user[""created_at""])
KeyError: 'created_at'
```
this looks awfully like #37 but it can't be, because i'm authed into my account and obviously i have perms to read my own account. wonder if there's any diagnostic methods i should apply here? just filing an issue for others to find while i investigate.",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed
1091850530,I_kwDODEm0Qs5BFFEi,63,Import archive error 'withheld_in_countries',521097,open,0,,,0,2022-01-01T16:58:59Z,2022-01-01T16:58:59Z,,NONE,,"Importing the twitter archive I received this error:
```bash
$ twitter-to-sqlite import archive.db twitter-2021-12-31-.zip
birdwatch-note-rating: not yet implemented
birdwatch-note: not yet implemented
branch-links: not yet implemented
community-tweet: not yet implemented
contact: not yet implemented
device-token: not yet implemented
direct-message-mute: not yet implemented
mute: not yet implemented
periscope-account-information: not yet implemented
periscope-ban-information: not yet implemented
periscope-broadcast-metadata: not yet implemented
periscope-comments-made-by-user: not yet implemented
periscope-expired-broadcasts: not yet implemented
periscope-followers: not yet implemented
periscope-profile-description: not yet implemented
professional-data: not yet implemented
protected-history: not yet implemented
reply-prompt: not yet implemented
screen-name-change: not yet implemented
smartblock: not yet implemented
spaces-metadata: not yet implemented
sso: not yet implemented
Traceback (most recent call last):
File ""/home/paulox/.virtualenvs/dogsheep/bin/twitter-to-sqlite"", line 8, in
sys.exit(cli())
File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py"", line 1128, in __call__
return self.main(*args, **kwargs)
File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py"", line 1053, in main
rv = self.invoke(ctx)
File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py"", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py"", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py"", line 754, in invoke
return __callback(*args, **kwargs)
File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/cli.py"", line 759, in import_
archive.import_from_file(db, filename, content)
File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/archive.py"", line 246, in import_from_file
db[table_name].insert_all(rows, pk=pk, replace=True)
File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2625, in insert_all
self.insert_chunk(
File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2406, in insert_chunk
result = self.db.execute(query, params)
File ""/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py"", line 422, in execute
return self.conn.execute(sql, parameters)
sqlite3.OperationalError: table archive_tweet has no column named withheld_in_countries
```
I found only a single tweet with the key `withheld_in_countries` in `tweet.js` that seems the problems:
```JSON
[
{
""tweet"" : {
""retweeted"" : false,
""source"" : ""Twitter for Android"",
""entities"" : {
""hashtags"" : [
{
""text"" : ""NowOnAndroid"",
""indices"" : [
""64"",
""77""
]
}
],
""symbols"" : [ ],
""user_mentions"" : [
{
""name"" : ""Periscope"",
""screen_name"" : ""PeriscopeCo"",
""indices"" : [
""3"",
""15""
],
""id_str"" : ""1111111111"",
""id"" : ""222222222""
}
],
""urls"" : [
{
""url"" : ""https://t.co/xxxxxxxxx"",
""expanded_url"" : ""https://vine.co/v/xxxxxxxxx"",
""display_url"" : ""vine.co/v/xxxxxxxxxx"",
""indices"" : [
""78"",
""101""
]
}
]
},
""display_text_range"" : [
""0"",
""101""
],
""favorite_count"" : ""0"",
""id_str"" : ""1111111111111111111111"",
""truncated"" : false,
""retweet_count"" : ""0"",
""withheld_in_countries"" : [
""TR""
],
""id"" : ""000000000000000000"",
""possibly_sensitive"" : false,
""created_at"" : ""Fri Aug 14 06:04:03 +0000 2015"",
""favorited"" : false,
""full_text"" : ""RT @periscopeco: Travel the world. LIVE. The Global Map is here #NowOnAndroid https://t.co/NZXdsPWROk"",
""lang"" : ""en""
}
}
]
```
I solved the error removing the key from the `tweet.js` but I'm reporting this error to improve the project.",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/63/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1097332098,I_kwDODEm0Qs5BZ_WC,64,Include all entities for tweets,111631,open,0,,,0,2022-01-09T23:35:28Z,2022-01-09T23:35:28Z,,NONE,,"Per our conversation [on Twitter](https://twitter.com/mschoening/status/1480312477246054401):
It would be neat if all entities (including URLs) were captured. This way you can ensure, that URLs are parsed out exactly the same way Twitter parses URLs – we all know parsing URLs with a regex ain't fun.
Right now, I believe the tool filters out all entities that are not of type `media`.",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/64/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1524431805,I_kwDODEm0Qs5a3Pu9,72,"Import thread, including self- and others' replies",601708,open,0,,,0,2023-01-08T09:51:06Z,2023-01-08T09:51:06Z,,NONE,,"statuses-lookup, home-timeline, mentions (only for auth'ed user) don't cover this.
`twitter-to-sqlite fetch-thread tw-group1.db 1234123412341234`
twitter-to-sqlite focuses on archiving users, but does not easily support archiving conversations or community activity.
For reference, this is [implemented in twarc](https://sourcegraph.com/github.com/DocNow/twarc/-/blob/twarc/client.py?L708-766&subtree=true), using a search, optionally recursively.
Other research suggests that this formerly, or currently, requires a [search query](https://stackoverflow.com/a/30480103/1020467), use of [undocumented `related_results` api](https://stackoverflow.com/a/9419346/1020467), or with requested inclusion of [newer conversation_id](https://stackoverflow.com/a/68115718/1020467) with subsequent query.
",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/72/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,
1816830546,I_kwDODEm0Qs5sSqJS,73,Twitter v1 API shutdown,6341745,open,0,,,0,2023-07-22T16:57:41Z,2023-07-22T16:57:41Z,,NONE,,"I've been using this project reliably over the past two years to periodically download my liked tweets, but unfortunately since 19th July I get:
```
[2023-07-19 21:00:04.937536] File ""/home/pi/code/liked-tweets/lib/python3.7/site-packages/twitter_to_sqlite/utils.py"", line 202, in fetch_timeline
[2023-07-19 21:00:04.937606] raise Exception(str(tweets[""errors""]))
[2023-07-19 21:00:04.937678] Exception: [{'message': 'You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level. You can learn more here: https://developer.twitter.com/en/portal/product', 'code': 453}]
```
It appears like Twitter has now shut down their v1 endpoints, which is rather gracious of them, considering they [announced they'd be deprecated on 29th April](https://twittercommunity.com/t/reminder-to-migrate-to-the-new-free-basic-or-enterprise-plans-of-the-twitter-api/189737).
Unfortunately [retrieving likes using the v2 API](https://developer.twitter.com/en/docs/twitter-api/tweets/likes/introduction) is not part of their [free plan](https://developer.twitter.com/en/portal/products). In fact, with the free plan one can only post and delete tweets and retrieve information about oneself.
So I'm afraid this is the end of this very nice project. It was very useful, thank you!
",206156866,issue,,,"{""url"": ""https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/73/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",,