{"id": 1816830546, "node_id": "I_kwDODEm0Qs5sSqJS", "number": 73, "title": "Twitter v1 API shutdown", "user": {"value": 6341745, "label": "david-perez"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-07-22T16:57:41Z", "updated_at": "2023-07-22T16:57:41Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I've been using this project reliably over the past two years to periodically download my liked tweets, but unfortunately since 19th July I get:\r\n\r\n```\r\n[2023-07-19 21:00:04.937536] File \"/home/pi/code/liked-tweets/lib/python3.7/site-packages/twitter_to_sqlite/utils.py\", line 202, in fetch_timeline\r\n[2023-07-19 21:00:04.937606] raise Exception(str(tweets[\"errors\"]))\r\n[2023-07-19 21:00:04.937678] Exception: [{'message': 'You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level. You can learn more here: https://developer.twitter.com/en/portal/product', 'code': 453}]\r\n```\r\n\r\nIt appears like Twitter has now shut down their v1 endpoints, which is rather gracious of them, considering they [announced they'd be deprecated on 29th April](https://twittercommunity.com/t/reminder-to-migrate-to-the-new-free-basic-or-enterprise-plans-of-the-twitter-api/189737).\r\n\r\nUnfortunately [retrieving likes using the v2 API](https://developer.twitter.com/en/docs/twitter-api/tweets/likes/introduction) is not part of their [free plan](https://developer.twitter.com/en/portal/products). In fact, with the free plan one can only post and delete tweets and retrieve information about oneself.\r\n\r\nSo I'm afraid this is the end of this very nice project. It was very useful, thank you!\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/73/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "draft": null, "state_reason": null}
{"id": 1524431805, "node_id": "I_kwDODEm0Qs5a3Pu9", "number": 72, "title": "Import thread, including self- and others' replies", "user": {"value": 601708, "label": "mcint"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-08T09:51:06Z", "updated_at": "2023-01-08T09:51:06Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "statuses-lookup, home-timeline, mentions (only for auth'ed user) don't cover this.\r\n\r\n`twitter-to-sqlite fetch-thread tw-group1.db 1234123412341234`\r\n\r\ntwitter-to-sqlite focuses on archiving users, but does not easily support archiving conversations or community activity.\r\n\r\nFor reference, this is [implemented in twarc](https://sourcegraph.com/github.com/DocNow/twarc/-/blob/twarc/client.py?L708-766&subtree=true), using a search, optionally recursively.\r\n\r\nOther research suggests that this formerly, or currently, requires a [search query](https://stackoverflow.com/a/30480103/1020467), use of [undocumented `related_results` api](https://stackoverflow.com/a/9419346/1020467), or with requested inclusion of [newer conversation_id](https://stackoverflow.com/a/68115718/1020467) with subsequent query.\r\n\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/72/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1077560091, "node_id": "I_kwDODEm0Qs5AOkMb", "number": 61, "title": "Data Pull fails for \"Essential\" level access to the Twitter API (for Documentation)", "user": {"value": 57161638, "label": "jmnickerson05"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-11T14:59:41Z", "updated_at": "2022-10-31T14:47:58Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Per Twitter documentation:\r\nhttps://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-leve\r\n\r\nThis isn't any fault of twitter-to-sqlite of course, but it should probably be documented as a side-note.\r\n\r\n\r\n\r\nAnd this is how I'm surfacing the message from utils.py:\r\n\r\n", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/61/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1063982712, "node_id": "I_kwDODEm0Qs4_axZ4", "number": 60, "title": "Execution on Windows", "user": {"value": 1733616, "label": "bernard01"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-11-26T00:24:34Z", "updated_at": "2022-10-14T16:58:27Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "My installation on Windows using pip has been successful. I have Python 3.6.\r\n\r\nHow do I run twitter-to-sqlite? I cannot even figure out how \"auth\" is a command. I have python on my path:\r\nC:\\prog\\python\\Python36;C:\\prog\\python\\Python36\\Scripts\r\n\r\nWhere should the commands be executed, and where are the files created?\r\n\r\nCould some basics please be added to the documentation to get beginners started?", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/60/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1088816961, "node_id": "I_kwDODEm0Qs5A5gdB", "number": 62, "title": "KeyError: 'created_at' for private accounts?", "user": {"value": 6764957, "label": "swyxio"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-12-26T17:51:51Z", "updated_at": "2022-03-12T02:36:32Z", "closed_at": "2022-02-24T18:10:18Z", "author_association": "NONE", "pull_request": null, "body": "hey Simon!\r\n\r\ni was running `twitter-to-sqlite user-timeline twitter.db` for [my private alt](https://twitter.com/swyxio) and ran into this error:\r\n\r\n\r\n\r\n\r\n\r\n\r\n
\r\n\r\n\r\n\r\n```bash\r\nTraceback (most recent call last):\r\n File \"/Users/swyx/Work/datasette/env/bin/twitter-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/cli.py\", line 291, in user_timeline\r\n profile = utils.get_profile(db, session, **kwargs)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py\", line 133, in get_profile\r\n save_users(db, [profile])\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py\", line 453, in save_users\r\n transform_user(user)\r\n File \"/Users/swyx/Work/datasette/env/lib/python3.9/site-packages/twitter_to_sqlite/utils.py\", line 285, in transform_user\r\n user[\"created_at\"] = parser.parse(user[\"created_at\"])\r\nKeyError: 'created_at'\r\n```\r\n\r\n \r\n\r\n\r\nthis looks awfully like #37 but it can't be, because i'm authed into my account and obviously i have perms to read my own account. wonder if there's any diagnostic methods i should apply here? just filing an issue for others to find while i investigate.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/62/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1097332098, "node_id": "I_kwDODEm0Qs5BZ_WC", "number": 64, "title": "Include all entities for tweets", "user": {"value": 111631, "label": "max"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-09T23:35:28Z", "updated_at": "2022-01-09T23:35:28Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Per our conversation [on Twitter](https://twitter.com/mschoening/status/1480312477246054401):\r\n\r\nIt would be neat if all entities (including URLs) were captured. This way you can ensure, that URLs are parsed out exactly the same way Twitter parses URLs \u2013 we all know parsing URLs with a regex ain't fun.\r\n\r\nRight now, I believe the tool filters out all entities that are not of type `media`.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/64/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1091850530, "node_id": "I_kwDODEm0Qs5BFFEi", "number": 63, "title": "Import archive error 'withheld_in_countries'", "user": {"value": 521097, "label": "pauloxnet"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-01T16:58:59Z", "updated_at": "2022-01-01T16:58:59Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "Importing the twitter archive I received this error:\r\n```bash\r\n$ twitter-to-sqlite import archive.db twitter-2021-12-31-.zip \r\nbirdwatch-note-rating: not yet implemented\r\nbirdwatch-note: not yet implemented\r\nbranch-links: not yet implemented\r\ncommunity-tweet: not yet implemented\r\ncontact: not yet implemented\r\ndevice-token: not yet implemented\r\ndirect-message-mute: not yet implemented\r\nmute: not yet implemented\r\nperiscope-account-information: not yet implemented\r\nperiscope-ban-information: not yet implemented\r\nperiscope-broadcast-metadata: not yet implemented\r\nperiscope-comments-made-by-user: not yet implemented\r\nperiscope-expired-broadcasts: not yet implemented\r\nperiscope-followers: not yet implemented\r\nperiscope-profile-description: not yet implemented\r\nprofessional-data: not yet implemented\r\nprotected-history: not yet implemented\r\nreply-prompt: not yet implemented\r\nscreen-name-change: not yet implemented\r\nsmartblock: not yet implemented\r\nspaces-metadata: not yet implemented\r\nsso: not yet implemented\r\nTraceback (most recent call last):\r\n File \"/home/paulox/.virtualenvs/dogsheep/bin/twitter-to-sqlite\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/cli.py\", line 759, in import_\r\n archive.import_from_file(db, filename, content)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/archive.py\", line 246, in import_from_file\r\n db[table_name].insert_all(rows, pk=pk, replace=True)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2625, in insert_all\r\n self.insert_chunk(\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2406, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py\", line 422, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.OperationalError: table archive_tweet has no column named withheld_in_countries\r\n```\r\n\r\nI found only a single tweet with the key `withheld_in_countries` in `tweet.js` that seems the problems:\r\n```JSON\r\n[\r\n{\r\n \"tweet\" : {\r\n \"retweeted\" : false,\r\n \"source\" : \"Twitter for Android\",\r\n \"entities\" : {\r\n \"hashtags\" : [\r\n {\r\n \"text\" : \"NowOnAndroid\",\r\n \"indices\" : [\r\n \"64\",\r\n \"77\"\r\n ]\r\n }\r\n ],\r\n \"symbols\" : [ ],\r\n \"user_mentions\" : [\r\n {\r\n \"name\" : \"Periscope\",\r\n \"screen_name\" : \"PeriscopeCo\",\r\n \"indices\" : [\r\n \"3\",\r\n \"15\"\r\n ],\r\n \"id_str\" : \"1111111111\",\r\n \"id\" : \"222222222\"\r\n }\r\n ],\r\n \"urls\" : [\r\n {\r\n \"url\" : \"https://t.co/xxxxxxxxx\",\r\n \"expanded_url\" : \"https://vine.co/v/xxxxxxxxx\",\r\n \"display_url\" : \"vine.co/v/xxxxxxxxxx\",\r\n \"indices\" : [\r\n \"78\",\r\n \"101\"\r\n ]\r\n }\r\n ]\r\n },\r\n \"display_text_range\" : [\r\n \"0\",\r\n \"101\"\r\n ],\r\n \"favorite_count\" : \"0\",\r\n \"id_str\" : \"1111111111111111111111\",\r\n \"truncated\" : false,\r\n \"retweet_count\" : \"0\",\r\n \"withheld_in_countries\" : [\r\n \"TR\"\r\n ],\r\n \"id\" : \"000000000000000000\",\r\n \"possibly_sensitive\" : false,\r\n \"created_at\" : \"Fri Aug 14 06:04:03 +0000 2015\",\r\n \"favorited\" : false,\r\n \"full_text\" : \"RT @periscopeco: Travel the world. LIVE. The Global Map is here #NowOnAndroid https://t.co/NZXdsPWROk\",\r\n \"lang\" : \"en\"\r\n }\r\n }\r\n ]\r\n```\r\n\r\nI solved the error removing the key from the `tweet.js` but I'm reporting this error to improve the project.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/63/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 796736607, "node_id": "MDU6SXNzdWU3OTY3MzY2MDc=", "number": 56, "title": "Not all quoted statuses get fetched?", "user": {"value": 42315895, "label": "gsajko"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-01-29T09:48:44Z", "updated_at": "2021-02-03T10:36:36Z", "closed_at": "2021-02-03T10:36:36Z", "author_association": "NONE", "pull_request": null, "body": "\r\n\r\n\r\nIn my database I have 13300 quote tweets, but eta 3600 have `quoted_status` empty.\r\n\r\nI fetched some of them using `https://api.twitter.com/1.1/statuses/show.json?id=xx` and they did have ids of quoted tweets.", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 771324837, "node_id": "MDU6SXNzdWU3NzEzMjQ4Mzc=", "number": 53, "title": "--since support for favorites", "user": {"value": 27, "label": "anotherjesse"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-12-19T07:08:23Z", "updated_at": "2020-12-19T07:47:11Z", "closed_at": "2020-12-19T07:47:11Z", "author_association": "NONE", "pull_request": null, "body": "Having support for `--since` for updating your favorites would be ideal as the api is both slow and it only returns ~3k most recent favorites.\r\n\r\nhttps://twittercommunity.com/t/cant-get-all-favorite-tweets-by-rest-api/22007/3\r\n\r\nThe api seems to take an optional `since_id` parameter - https://developer.twitter.com/en/docs/twitter-api/v1/tweets/post-and-engage/api-reference/get-favorites-list", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 745393298, "node_id": "MDU6SXNzdWU3NDUzOTMyOTg=", "number": 52, "title": "Discussion: Adding support for fetching only fresh tweets", "user": {"value": 4169772, "label": "fatihky"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-18T07:01:48Z", "updated_at": "2020-11-18T07:12:45Z", "closed_at": "2020-11-18T07:12:45Z", "author_association": "NONE", "pull_request": null, "body": "I think it'd be very useful if this tool has an option like `--incremental` to fetch only newer tweets. This way operations could complete very fast in sequential runs. I'd want to try to implement this feature if it seems OK for this tool's purpose. ", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 639542974, "node_id": "MDU6SXNzdWU2Mzk1NDI5NzQ=", "number": 47, "title": "Fall back to FTS4 if FTS5 is not available", "user": {"value": 73579, "label": "hpk42"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-06-16T10:11:23Z", "updated_at": "2020-06-17T20:13:48Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "got this with version 0.21.1 from pypi. twitter-to-sqlite auth worked but then \"twitter-to-sqlite user-timeline USER.db\" produced a tracekback ending in \"no such module: FTS5\". ", "repo": {"value": 206156866, "label": "twitter-to-sqlite"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}