github
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
603624862 | MDU6SXNzdWU2MDM2MjQ4NjI= | 31 | Issue and milestone should have foreign key to repo | 9599 | closed | 0 | 3 | 2020-04-21T00:46:24Z | 2020-04-22T01:20:19Z | 2020-04-22T01:20:19Z | MEMBER | Currently the `repo` column on those tables is a string `simonw/datasette` rather than an ID referencing a row in `repos`. _Originally posted by @simonw in https://github.com/dogsheep/github-to-sqlite/issues/29#issuecomment-616883275_ | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/31/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
604222295 | MDU6SXNzdWU2MDQyMjIyOTU= | 32 | Issue comments don't appear to populate issues foreign key | 9599 | closed | 0 | 3 | 2020-04-21T19:17:32Z | 2020-04-22T01:17:44Z | 2020-04-22T01:17:44Z | MEMBER | https://github-to-sqlite.dogsheep.net/github?sql=select+html_url%2C+id%2C+issue+from+issue_comments+order+by+updated_at+desc+limit+101 <img width="774" alt="Screen Shot 2020-04-21 at 12 16 57 PM" src="https://user-images.githubusercontent.com/9599/79904752-0b091b00-83ca-11ea-9b15-b2893f05dfdc.png"> | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/32/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
610408908 | MDU6SXNzdWU2MTA0MDg5MDg= | 34 | Command for retrieving dependents for a repo | 9599 | closed | 0 | 6 | 2020-04-30T21:47:51Z | 2020-05-03T15:53:01Z | 2020-05-03T15:53:01Z | MEMBER | I really, really want to start grabbing this data: https://github.com/simonw/datasette/network/dependents | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/34/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
610511450 | MDU6SXNzdWU2MTA1MTE0NTA= | 35 | Create index on issue_comments(user) and other foreign keys | 9599 | closed | 0 | 3 | 2020-05-01T02:06:56Z | 2020-05-02T18:26:24Z | 2020-05-02T18:26:24Z | MEMBER | ``` create index issue_comments_user on issue_comments(user) ``` I'm sure there are other user columns that could benefit from an index. | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/35/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
610842926 | MDU6SXNzdWU2MTA4NDI5MjY= | 36 | Add view for better display of dependent repos | 9599 | closed | 0 | 2 | 2020-05-01T16:33:44Z | 2020-05-02T16:50:31Z | 2020-05-02T16:30:11Z | MEMBER | ```sql select repos.full_name as repo, 'https://github.com/' || repos2.full_name as dependent, repos2.created_at as dependent_repo_created, repos2.updated_at as dependent_repo_updated, repos2.stargazers_count as dependent_repo_stars, repos2.watchers_count as dependent_repo_watchers from dependents join repos as repos2 on dependents.dependent = repos2.id join repos on dependents.repo = repos.id order by repos2.created_at desc ``` https://dogsheep.simonwillison.net/github?sql=select%0D%0A++repos.full_name+as+repo%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos2.full_name+as+dependent%2C%0D%0A++repos2.created_at+as+dependent_repo_created%2C%0D%0A++repos2.updated_at+as+dependent_repo_updated%2C%0D%0A++repos2.stargazers_count+as+dependent_repo_stars%2C%0D%0A++repos2.watchers_count+as+dependent_repo_watchers%0D%0Afrom%0D%0A++dependents%0D%0A++join+repos+as+repos2+on+dependents.dependent+%3D+repos2.id%0D%0A++join+repos+on+dependents.repo+%3D+repos.id%0D%0Aorder+by%0D%0A++repos2.created_at+desc | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/36/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
610843136 | MDU6SXNzdWU2MTA4NDMxMzY= | 37 | Mechanism for creating views if they don't yet exist | 9599 | closed | 0 | 3 | 2020-05-01T16:34:10Z | 2020-05-02T16:19:47Z | 2020-05-02T16:19:31Z | MEMBER | Needed for #36 #10 #12 | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/37/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
613777056 | MDU6SXNzdWU2MTM3NzcwNTY= | 39 | issues foreign key to repo isn't working | 9599 | closed | 0 | 1 | 2020-05-07T05:11:48Z | 2020-08-18T14:24:46Z | 2020-08-18T14:23:56Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/issues?_facet=repo <img width="725" alt="github__issues__2_303_rows_where_sorted_by_updated_at_descending" src="https://user-images.githubusercontent.com/9599/81256810-723be780-8fe6-11ea-9e1f-f37d40f4d3ae.png"> If the foreign key was working those would be repository names. From the schema at the bottom of the page: ``` [repo] TEXT, ``` That's the wrong type and not a foreign key. | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/39/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
493670730 | MDU6SXNzdWU0OTM2NzA3MzA= | 4 | Command to fetch stargazers for one or more repos | 9599 | closed | 0 | 8 | 2019-09-14T21:58:22Z | 2020-05-02T21:30:27Z | 2020-05-02T21:30:27Z | MEMBER | Maybe this: $ github-to-sqlite stargazers github.db simonw/datasette It could accept more than one repos. Maybe have options similar to `--sql` in [twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) so you can e.g. fetch all stargazers for all of the repos you have fetched into the database already (or all of the repos belonging to owner X) | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
637899539 | MDU6SXNzdWU2Mzc4OTk1Mzk= | 40 | Demo deploy is broken | 9599 | closed | 0 | 2 | 2020-06-12T17:20:17Z | 2020-06-12T18:06:48Z | 2020-06-12T18:06:48Z | MEMBER | https://github.com/dogsheep/github-to-sqlite/runs/766180404?check_suite_focus=true ``` The following NEW packages will be installed: sqlite3 0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded. Need to get 752 kB of archives. After this operation, 2482 kB of additional disk space will be used. Ign:1 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3 Err:1 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 sqlite3 amd64 3.22.0-1ubuntu0.3 404 Not Found [IP: 52.177.174.250 80] E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/s/sqlite3/sqlite3_3.22.0-1ubuntu0.3_amd64.deb 404 Not Found [IP: 52.177.174.250 80] E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? ##[error]Process completed with exit code 100. ``` | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/40/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
651159727 | MDU6SXNzdWU2NTExNTk3Mjc= | 41 | Demo is failing to deploy | 9599 | closed | 0 | 7 | 2020-07-05T22:40:33Z | 2020-07-06T01:07:03Z | 2020-07-06T01:07:02Z | MEMBER | https://github.com/dogsheep/github-to-sqlite/runs/837714622?check_suite_focus=true ``` Creating Revision.........................................................................................................................................failed Deployment failed ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information. Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette", line 8, in <module> sys.exit(cli()) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py", line 138, in cloudrun check_call( File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'gcloud run deploy --allow-unauthenticated --platform=managed --image gcr.io/datasette-222320/datasette github-to-sqlite' returned non-zero exit sta… | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
654405302 | MDU6SXNzdWU2NTQ0MDUzMDI= | 42 | Option for importing just specific repos | 9599 | closed | 0 | 0 | 2020-07-09T23:20:15Z | 2020-07-09T23:25:35Z | 2020-07-09T23:25:35Z | MEMBER | For if you know which specific repos you care about, as opposed to loading everything owned by the authenticated user. github-to-sqlite repos specific.db -r simonw/datasette -r simonw/github-contents | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/42/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
660355904 | MDU6SXNzdWU2NjAzNTU5MDQ= | 43 | github-to-sqlite tags command for fetching tags | 9599 | closed | 0 | 4 | 2020-07-18T20:14:12Z | 2020-07-18T23:05:56Z | 2020-07-18T21:52:15Z | MEMBER | Fetches paginated data from https://api.github.com/repos/simonw/datasette/tags | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
660413281 | MDU6SXNzdWU2NjA0MTMyODE= | 44 | Rename tags.repo_id column to tags.repo | 9599 | closed | 0 | 0 | 2020-07-18T22:13:46Z | 2020-07-18T22:15:12Z | 2020-07-18T22:15:12Z | MEMBER | For improved consistency with other tables. https://observablehq.com/@simonw/datasette-table-diagram ![datasette-table-diagram(1)](https://user-images.githubusercontent.com/9599/87862843-3cca4900-c909-11ea-9c76-58b3f4aca43f.png) | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/44/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
660429601 | MDU6SXNzdWU2NjA0Mjk2MDE= | 45 | Fix the demo - it breaks because of the tags table change | 9599 | closed | 0 | 5 | 2020-07-18T22:49:32Z | 2020-07-18T23:03:14Z | 2020-07-18T23:03:13Z | MEMBER | https://github.com/dogsheep/github-to-sqlite/runs/885773677 ``` File "/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py", line 476, in save_tags db["tags"].insert_all( File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/sqlite_utils/db.py", line 1145, in insert_all result = self.db.conn.execute(query, params) sqlite3.OperationalError: table tags has no column named repo ``` That's because I changed the name in #44. I thought this would be safe since no-one else could possibly be using this yet (it hadn't shipped in a release) but turns out I broke my demo! | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
681086659 | MDU6SXNzdWU2ODEwODY2NTk= | 47 | emojis command | 9599 | closed | 0 | 1 | 2020-08-18T14:26:26Z | 2020-08-18T14:52:13Z | 2020-08-18T14:52:13Z | MEMBER | For fun - it can import https://api.github.com/emojis - maybe with an option to fetch the binary representations in addition to the URLs. | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/47/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
493671014 | MDU6SXNzdWU0OTM2NzEwMTQ= | 5 | Add "incomplete" boolean to users table for incomplete profiles | 9599 | closed | 0 | 2 | 2019-09-14T22:01:50Z | 2020-03-23T19:23:31Z | 2020-03-23T19:23:30Z | MEMBER | User profiles that are fetched from e.g. stargazers (#4) are incomplete - they have a login but they don't have name, company etc. Add a `incomplete` boolean flag to the `users` table to record this. Then later I can add a `backfill-users` command which loops through and fetches missing data for those incomplete profiles. | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
724264574 | MDU6SXNzdWU3MjQyNjQ1NzQ= | 52 | Option to fetch README and/or HTML-rendered README for repos | 9599 | closed | 0 | 0 | 2020-10-19T05:10:24Z | 2020-10-19T05:33:42Z | 2020-10-19T05:33:42Z | MEMBER | I'm thinking: github-to-sqlite repos ... --readme # Populates readme column with raw text github-to-sqlite repos ... --readme-html # Populates readme_html column with raw HTML https://developer.github.com/v3/repos/contents/#get-a-repository-readme | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/52/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
753026003 | MDU6SXNzdWU3NTMwMjYwMDM= | 54 | github-to-sqlite workflows command | 9599 | closed | 0 | 3 | 2020-11-29T21:56:42Z | 2020-11-29T22:08:46Z | 2020-11-29T21:57:17Z | MEMBER | A command that fetches the YAML workflows for different repos, parses them and stores them in relational tables would be really useful for maintaining larger numbers of workflows. | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
753026388 | MDU6SXNzdWU3NTMwMjYzODg= | 55 | github-to-sqlite workflows does not correctly replace existing records | 9599 | closed | 0 | 0 | 2020-11-29T21:58:43Z | 2020-11-29T23:48:50Z | 2020-11-29T23:48:50Z | MEMBER | Following #54 - see this TODO: https://github.com/dogsheep/github-to-sqlite/blob/1b23ce11953f9f59c0161ea1f99188b55b5ea11c/github_to_sqlite/utils.py#L700 | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/55/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
753122082 | MDU6SXNzdWU3NTMxMjIwODI= | 56 | Link to example tables from the README | 9599 | closed | 0 | 0 | 2020-11-30T04:01:51Z | 2020-11-30T04:10:27Z | 2020-11-30T04:10:27Z | MEMBER | Would help demonstrate how the tool works. | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/56/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
758944006 | MDU6SXNzdWU3NTg5NDQwMDY= | 57 | --readme throws 404 error if README does not exist in repo | 9599 | closed | 0 | 0 | 2020-12-07T23:58:49Z | 2020-12-16T18:17:54Z | 2020-12-16T18:17:54Z | MEMBER | It should fail silently (populate the column with a null) instead. | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/57/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
769150394 | MDU6SXNzdWU3NjkxNTAzOTQ= | 58 | Readme HTML has broken internal links | 9599 | closed | 0 | 2 | 2020-12-16T17:58:11Z | 2020-12-16T19:20:14Z | 2020-12-16T19:20:14Z | MEMBER | From https://github.com/simonw/datasette.io/issues/46 ```html <li><a href="#filtering-tables">Filtering tables</a></li> ... <h3><a id="user-content-filtering-tables" class="anchor" aria-hidden="true" href="#filtering-tables"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Filtering tables</h3> ``` So this is a bug in GitHub's API, but we need to work around it. | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
506276893 | MDU6SXNzdWU1MDYyNzY4OTM= | 7 | issue-comments command for importing issue comments | 9599 | closed | 0 | 1 | 2019-10-13T05:23:58Z | 2019-10-14T14:44:12Z | 2019-10-13T05:24:30Z | MEMBER | Using this API: https://developer.github.com/v3/issues/comments/ | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1308461063 | I_kwDODFdgUs5N_YgH | 74 | 500 error in github-to-sqlite demo | 9599 | closed | 0 | 5 | 2022-07-18T19:39:32Z | 2022-07-18T21:16:18Z | 2022-07-18T21:14:22Z | MEMBER | https://github-to-sqlite.dogsheep.net/github/issue_comments throws a 500: > `cannot import name 'etree' from 'markdown.util' (/usr/local/lib/python3.8/site-packages/markdown/util.py)` https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/metrics?project=datasette-222320 suggests this started happening 3 days ago. | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
516763727 | MDExOlB1bGxSZXF1ZXN0MzM1OTgwMjQ2 | 8 | stargazers command, refs #4 | 9599 | closed | 0 | 5 | 2019-11-03T00:37:36Z | 2020-05-02T20:00:27Z | 2020-05-02T20:00:26Z | MEMBER | dogsheep/github-to-sqlite/pulls/8 | Needs tests. Refs #4. | 207052882 | pull | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
516769276 | MDU6SXNzdWU1MTY3NjkyNzY= | 9 | Commands do not work without an auth.json file | 9599 | closed | 0 | 0 | 2019-11-03T01:54:28Z | 2019-11-11T05:30:48Z | 2019-11-11T05:30:48Z | MEMBER | `auth.json` is meant to be optional. If it's not provided, the tool should make heavily rate-limited unauthenticated requests. ``` $ github-to-sqlite repos .data/repos.db simonw Usage: github-to-sqlite repos [OPTIONS] DB_PATH [USERNAME] Try "github-to-sqlite repos --help" for help. Error: Invalid value for "-a" / "--auth": File "auth.json" does not exist. ``` | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
585526292 | MDU6SXNzdWU1ODU1MjYyOTI= | 1 | Set up full text search | 9599 | closed | 0 | 1 | 2020-03-21T15:57:35Z | 2020-03-21T19:47:46Z | 2020-03-21T19:45:52Z | MEMBER | Should run against `title` and `text` in `items`, and `about` and `id` in `users`. | 248903544 | issue | { "url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
470637068 | MDU6SXNzdWU0NzA2MzcwNjg= | 1 | Use XML Analyser to figure out the structure of the export XML | 9599 | closed | 0 | 1 | 2019-07-20T05:19:02Z | 2019-07-20T05:20:09Z | 2019-07-20T05:20:09Z | MEMBER | https://github.com/simonw/xml_analyser | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
519038979 | MDU6SXNzdWU1MTkwMzg5Nzk= | 10 | Failed to import workout points | 9599 | closed | 0 | 4 | 2019-11-07T04:50:22Z | 2019-11-08T01:18:37Z | 2019-11-08T01:18:37Z | MEMBER | I just ran the script and it failed to import any `workout_points`, though it did import `workouts`. | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
975158266 | MDU6SXNzdWU5NzUxNTgyNjY= | 19 | table activity_summary has no column named appleMoveTime | 9599 | closed | 0 | 0 | 2021-08-20T00:46:44Z | 2021-08-20T00:54:34Z | 2021-08-20T00:54:34Z | MEMBER | Got this error today against a fresh export: table activity_summary has no column named appleMoveTime | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
470637152 | MDU6SXNzdWU0NzA2MzcxNTI= | 2 | Import workouts | 9599 | closed | 0 | 1 | 2019-07-20T05:20:21Z | 2019-07-20T06:21:41Z | 2019-07-20T06:21:41Z | MEMBER | From #1 | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
470637206 | MDU6SXNzdWU0NzA2MzcyMDY= | 3 | Import ActivitySummary | 9599 | closed | 0 | 0 | 2019-07-20T05:21:00Z | 2019-07-20T05:58:07Z | 2019-07-20T05:58:07Z | MEMBER | From #1 ```python 'ActivitySummary': {'attr_counts': {'activeEnergyBurned': 980, 'activeEnergyBurnedGoal': 980, 'activeEnergyBurnedUnit': 980, 'appleExerciseTime': 980, 'appleExerciseTimeGoal': 980, 'appleStandHours': 980, 'appleStandHoursGoal': 980, 'dateComponents': 980}, 'child_counts': {}, 'count': 980, 'parent_counts': {'HealthData': 980}}, ``` | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
470640505 | MDU6SXNzdWU0NzA2NDA1MDU= | 4 | Import Records | 9599 | closed | 0 | 1 | 2019-07-20T06:11:20Z | 2019-07-20T06:21:41Z | 2019-07-20T06:21:41Z | MEMBER | From #1: ```python 'Record': {'attr_counts': {'creationDate': 2672233, 'device': 2665111, 'endDate': 2672233, 'sourceName': 2672233, 'sourceVersion': 2671779, 'startDate': 2672233, 'type': 2672233, 'unit': 2650012, 'value': 2672232}, 'child_counts': {'HeartRateVariabilityMetadataList': 2318, 'MetadataEntry': 287974}, 'count': 2672233, 'parent_counts': {'Correlation': 2, 'HealthData': 2672231}}, ``` | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
470691622 | MDU6SXNzdWU0NzA2OTE2MjI= | 5 | Add progress bar | 9599 | closed | 0 | 2 | 2019-07-20T16:29:07Z | 2019-07-22T03:30:13Z | 2019-07-22T02:49:22Z | MEMBER | Showing a progress bar would be nice, using Click. The easiest way to do this would probably be be to hook it up to the length of the compressed content, and update it as this code pushes more XML bytes through the parser: https://github.com/dogsheep/healthkit-to-sqlite/blob/d64299765064501f4efdd9a0b21dbdba9ec4287f/healthkit_to_sqlite/utils.py#L6-L10 | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
470856782 | MDU6SXNzdWU0NzA4NTY3ODI= | 6 | Break up records into different tables for each type | 9599 | closed | 0 | 1 | 2019-07-22T01:54:59Z | 2019-07-22T03:28:55Z | 2019-07-22T03:28:50Z | MEMBER | I don't think there's much benefit to having all of the different record types stored in the same enormous table. Here's what I get when I use `_facet=type`: <img width="358" alt="hello2__records__2_672_233_rows" src="https://user-images.githubusercontent.com/9599/61601118-e2f54d00-abe8-11e9-8bf6-3df2ef969112.png"> I'm going to try splitting these up into separate tables - so `HKQuantityTypeIdentifierBodyMassIndex` becomes a table called `rBodyMassIndex` - and see if that's nicer to work with. | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
472097220 | MDU6SXNzdWU0NzIwOTcyMjA= | 7 | Script uses a lot of RAM | 9599 | closed | 0 | 3 | 2019-07-24T06:11:11Z | 2019-07-24T06:35:52Z | 2019-07-24T06:35:52Z | MEMBER | I'm using an XML pull parser which should avoid the need to slurp the whole XML file into memory, but it's not working - the script still uses over 1GB of RAM when it runs according to Activity Monitor. I think this is because I'm still causing the full root element to be incrementally loaded into memory just in case I try and access it later. http://effbot.org/elementtree/iterparse.htm says I should use `elem.clear()` as I go. It also says: > The above pattern has one drawback; it does not clear the root element, so you will end up with a single element with lots of empty child elements. If your files are huge, rather than just large, this might be a problem. To work around this, you need to get your hands on the root element. So I will try that recipe and see if it helps. | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
472104705 | MDExOlB1bGxSZXF1ZXN0MzAwNTgwMjIx | 8 | Use less RAM | 9599 | closed | 0 | 0 | 2019-07-24T06:35:01Z | 2019-07-24T06:35:52Z | 2019-07-24T06:35:52Z | MEMBER | dogsheep/healthkit-to-sqlite/pulls/8 | Closes #7 | 197882382 | pull | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
503233021 | MDU6SXNzdWU1MDMyMzMwMjE= | 1 | Use better pagination (and implement progress bar) | 9599 | closed | 0 | 4 | 2019-10-07T04:58:11Z | 2020-03-27T22:13:57Z | 2020-03-27T22:13:57Z | MEMBER | Right now we attempt to load everything at once - which caps out at 5,000 items and is really slow. We can do better by implementing pagination using count and offset. | 213286752 | issue | { "url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1795187493 | I_kwDODLZ_YM5rAGMl | 12 | Switch to pyproject.toml | 9599 | closed | 0 | 2 | 2023-07-09T01:06:56Z | 2023-07-09T01:19:43Z | 2023-07-09T01:19:42Z | MEMBER | First of my CLI tools to use https://til.simonwillison.net/python/pyproject | 213286752 | issue | { "url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
503234169 | MDU6SXNzdWU1MDMyMzQxNjk= | 2 | Track and use the 'since' value | 9599 | closed | 0 | 3 | 2019-10-07T05:02:59Z | 2020-03-27T22:22:30Z | 2020-03-27T22:22:30Z | MEMBER | Pocket says: > Whenever possible, you should use the since parameter, or count and and offset parameters when retrieving a user's list. After retrieving the list, you should store the current time (which is provided along with the list response) and pass that in the next request for the list. This way the server only needs to return a small set (changes since that time) instead of the user's entire list every time. At the bottom of https://getpocket.com/developer/docs/v3/retrieve | 213286752 | issue | { "url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
589402939 | MDU6SXNzdWU1ODk0MDI5Mzk= | 4 | Store authentication information as "pocket_access_token" etc | 9599 | closed | 0 | 0 | 2020-03-27T20:43:22Z | 2020-03-27T20:43:59Z | 2020-03-27T20:43:59Z | MEMBER | The `pocket_` prefix will mean that the same `auth.json` file can be used for other Dogsheep tools without Pocket over-riding a value set by some other tool. | 213286752 | issue | { "url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
487598042 | MDU6SXNzdWU0ODc1OTgwNDI= | 1 | Implement code to pull checkins from the Foursquare API | 9599 | closed | 0 | 0 | 2019-08-30T17:40:02Z | 2019-08-30T18:23:24Z | 2019-08-30T18:23:24Z | MEMBER | The tool currently only works with a pre-prepared JSON file of checkins. When called without options, it should prompt the user to paste in a Foursquare OAuth token. The `--token=` option should work too, and should be backed up by an optional environment variable. | 205429375 | issue | { "url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1661617056 | I_kwDODD6af85jCkOg | 15 | ambiguous column name: createdAt - on checkin_details view | 9599 | closed | 0 | 0 | 2023-04-11T01:07:47Z | 2023-04-11T03:16:37Z | 2023-04-11T03:16:37Z | MEMBER | It looks like Swarm changed their schema and now both `venues` and `checkins` have `createdAt` fields. Which breaks this view: https://github.com/dogsheep/swarm-to-sqlite/blob/719b6e96a016d0ca8b316d3bed9c2a7a0cb499ee/swarm_to_sqlite/utils.py#L171-L188 | 205429375 | issue | { "url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
487598468 | MDU6SXNzdWU0ODc1OTg0Njg= | 2 | --save option to dump checkins to a JSON file on disk | 9599 | closed | 0 | 1 | 2019-08-30T17:41:06Z | 2019-08-31T02:40:21Z | 2019-08-31T02:40:21Z | MEMBER | This is a complement to the `--load` option - mainly useful for development purposes. (I'll rename `--file` to `--load` as part of this issue). | 205429375 | issue | { "url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
487600595 | MDU6SXNzdWU0ODc2MDA1OTU= | 3 | Option to fetch only checkins more recent than the current max checkin | 9599 | closed | 0 | 4 | 2019-08-30T17:46:45Z | 2019-10-16T20:41:23Z | 2019-10-16T20:39:59Z | MEMBER | The Foursquare checkins API supports "return every checkin occurring after this point" - I can pass it the maximum createdAt date currently stored in the database. This will allow for quick incremental fetches via a cron. | 205429375 | issue | { "url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
487601121 | MDU6SXNzdWU0ODc2MDExMjE= | 4 | Online tool for getting a Foursquare OAuth token | 9599 | closed | 0 | 1 | 2019-08-30T17:48:14Z | 2019-08-31T18:07:26Z | 2019-08-31T18:07:26Z | MEMBER | I will link to this from the documentation. See also this conversation on Twitter: https://twitter.com/simonw/status/1166822603023011840 I've decided to go with "copy and paste in a token" rather than hooking up a local web server that can have tokens passed to it. | 205429375 | issue | { "url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
487721884 | MDU6SXNzdWU0ODc3MjE4ODQ= | 5 | Treat Foursquare timestamps as UTC | 9599 | closed | 0 | 0 | 2019-08-31T02:44:47Z | 2019-08-31T02:50:41Z | 2019-08-31T02:50:41Z | MEMBER | Current test failure is due to timezone differences between my laptop and Circle CI: https://circleci.com/gh/dogsheep/swarm-to-sqlite/3 ``` E Full diff: E - [{'created': '2018-07-01T04:48:19', E ? ^ E + [{'created': '2018-07-01T02:48:19', E ? ^ E 'createdAt': 1530413299, ``` The timestamps I store in `created` should always be UTC. | 205429375 | issue | { "url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
589491711 | MDU6SXNzdWU1ODk0OTE3MTE= | 7 | Upgrade to sqlite-utils 2.x | 9599 | closed | 0 | 0 | 2020-03-28T02:24:51Z | 2020-03-28T02:25:03Z | 2020-03-28T02:25:03Z | MEMBER | 205429375 | issue | { "url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
488833136 | MDU6SXNzdWU0ODg4MzMxMzY= | 1 | Imported followers should go in "users", relationships in "following" | 9599 | closed | 0 | 0 | 2019-09-03T21:27:37Z | 2019-09-04T20:23:04Z | 2019-09-04T20:23:04Z | MEMBER | Right now `twitter-to-sqlite followers` dumps everything in a `followers` table, and doesn't actually record which account they are following! It should instead save them all in a global `users` table and then set up m2m relationships in a `following` table. This also means it should create a record for the specified user in order to record both sides of each relationship. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
492297930 | MDU6SXNzdWU0OTIyOTc5MzA= | 10 | Rethink progress bars for various commands | 9599 | closed | 0 | 5 | 2019-09-11T15:06:47Z | 2020-04-01T03:45:48Z | 2020-04-01T03:45:48Z | MEMBER | Progress bars and the `--silent` option are implemented inconsistently across commands at the moment. This is made more challenging by the fact that for many operations the total length is not known. https://click.palletsprojects.com/en/7.x/api/#click.progressbar | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
503045221 | MDU6SXNzdWU1MDMwNDUyMjE= | 11 | Commands for recording real-time tweets from the streaming API | 9599 | closed | 0 | 1 | 2019-10-06T03:09:30Z | 2019-10-06T04:54:17Z | 2019-10-06T04:48:31Z | MEMBER | https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter We can support tracking keywords and following specific users. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
503053800 | MDU6SXNzdWU1MDMwNTM4MDA= | 12 | Extract "source" into a separate lookup table | 9599 | closed | 0 | 3 | 2019-10-06T05:17:23Z | 2019-10-17T15:49:24Z | 2019-10-17T15:49:24Z | MEMBER | It's pretty bulky and ugly at the moment: <img width="334" alt="trump__tweets__1_820_rows" src="https://user-images.githubusercontent.com/9599/66264630-df23a080-e7bd-11e9-9154-403c2e69f841.png"> | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
503085013 | MDU6SXNzdWU1MDMwODUwMTM= | 13 | statuses-lookup command | 9599 | closed | 0 | 1 | 2019-10-06T11:00:20Z | 2019-10-07T00:33:49Z | 2019-10-07T00:31:44Z | MEMBER | For bulk retrieving tweets by their ID. https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-lookup Rate limit is 900/15 minutes (1 call per second) but each call can pull up to 100 IDs, so we can pull 6,000 per minute. Should support `--SQL` and `--attach` #8 | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
503244410 | MDU6SXNzdWU1MDMyNDQ0MTA= | 14 | When importing favorites, record which user favorited them | 9599 | closed | 0 | 0 | 2019-10-07T05:45:11Z | 2019-10-14T03:30:25Z | 2019-10-14T03:30:25Z | MEMBER | This code currently just dumps them into the `tweets` table without recording who it was who had favorited them. https://github.com/dogsheep/twitter-to-sqlite/blob/436a170d74ec70903d1b4ca430c2c6b6435cdfcc/twitter_to_sqlite/cli.py#L152-L157 | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/14/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
505666744 | MDExOlB1bGxSZXF1ZXN0MzI3MDUxNjcz | 15 | twitter-to-sqlite import command, refs #4 | 9599 | closed | 0 | 0 | 2019-10-11T06:37:14Z | 2019-10-11T06:45:01Z | 2019-10-11T06:45:01Z | MEMBER | dogsheep/twitter-to-sqlite/pulls/15 | 206156866 | pull | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
505674949 | MDU6SXNzdWU1MDU2NzQ5NDk= | 17 | import command should empty all archive-* tables first | 9599 | closed | 0 | 2 | 2019-10-11T06:58:43Z | 2019-10-11T15:40:08Z | 2019-10-11T15:40:08Z | MEMBER | Can have a CLI option for NOT doing that. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
505928530 | MDU6SXNzdWU1MDU5Mjg1MzA= | 18 | Command to import home-timeline | 9599 | closed | 0 | 4 | 2019-10-11T15:47:54Z | 2019-10-11T16:51:33Z | 2019-10-11T16:51:12Z | MEMBER | Feature request: https://twitter.com/johankj/status/1182563563136868352 > Would it be possible to save all tweets in my timeline from the last X days? I would love to see how big a percentage some users are of my daily timeline as a metric on whether I should unfollow them/move them to a list. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
506087267 | MDU6SXNzdWU1MDYwODcyNjc= | 19 | since_id support for home-timeline | 9599 | closed | 0 | 3 | 2019-10-11T22:48:24Z | 2019-10-16T19:13:06Z | 2019-10-16T19:12:46Z | MEMBER | Currently every time you run `home-timeline` we pull all 800 available tweets. We should offer to support `since_id` (which can be provided or can be pulled directly from the database) in order to work more efficiently if this command is executed e.g. on a cron. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
488833698 | MDU6SXNzdWU0ODg4MzM2OTg= | 2 | "twitter-to-sqlite user-timeline" command for pulling tweets by a specific user | 9599 | closed | 0 | 3 | 2019-09-03T21:29:12Z | 2019-09-04T20:02:11Z | 2019-09-04T20:02:11Z | MEMBER | Twitter only allows up to 3,200 tweets to be retrieved from https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html I'm going to do: $ twitter-to-sqlite tweets simonw | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
506268945 | MDU6SXNzdWU1MDYyNjg5NDU= | 20 | --since support for various commands for refresh-by-cron | 9599 | closed | 0 | 3 | 2019-10-13T03:40:46Z | 2019-10-21T03:32:04Z | 2019-10-16T19:26:11Z | MEMBER | I want to run a cron that updates my Twitter database every X minutes. It should be able to retrieve the following without needing to paginate through everything: - [x] Tweets I have tweeted - [x] My home timeline (see #19) - [x] Tweets I have favourited It would be nice if this could be standardized across all commands as a `--since` option. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
506432572 | MDU6SXNzdWU1MDY0MzI1NzI= | 21 | Fix & escapes in tweet text | 9599 | closed | 0 | 1 | 2019-10-14T03:37:28Z | 2019-10-15T18:48:16Z | 2019-10-15T18:48:16Z | MEMBER | <img width="1136" alt="twitter__tweets__21_773_rows_where_sorted_by_id_descending" src="https://user-images.githubusercontent.com/9599/66728360-38f91b80-edf9-11e9-95b5-ce6d097fe18e.png"> Shouldn't be storing `&` here. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/21/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
508024032 | MDU6SXNzdWU1MDgwMjQwMzI= | 22 | Ability to import from uncompressed archive or from specific files | 9599 | closed | 0 | 0 | 2019-10-16T18:31:57Z | 2019-10-16T18:53:36Z | 2019-10-16T18:53:36Z | MEMBER | Currently you can only import like this: $ twitter-to-sqlite import path-to-twitter.zip It would be useful if you could import from a folder that was decompressed from that zip: $ twitter-to-sqlite import path-to-twitter/ AND from individual files within that folder - since that would allow you to e.g. selectively import certain files: $ twitter-to-sqlite import path-to-twitter/favorites.js path-to-twitter/tweets.js | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/22/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
508190730 | MDU6SXNzdWU1MDgxOTA3MzA= | 23 | Extremely simple migration system | 9599 | closed | 0 | 2 | 2019-10-17T02:13:57Z | 2019-10-17T16:57:17Z | 2019-10-17T16:57:17Z | MEMBER | Needed for #12. This is going to be an incredibly simple version of the Django migration system. * A `migrations` table, keeping track of which migrations were applied (and when) * A `migrate()` function which applies any pending migrations * A `MIGRATIONS` constant which is a list of functions to be applied The function names will be detected and used as the names of the migrations. Every time you run the CLI tool it will call the `migrate()` function before doing anything else. Needs to take into account that there might be no tables at all. As such, migration functions should sanity check that the tables they are going to work on actually exist. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
508553387 | MDExOlB1bGxSZXF1ZXN0MzI5MzI0MzY4 | 24 | Tweet source extraction and new migration system | 9599 | closed | 0 | 0 | 2019-10-17T15:24:56Z | 2019-10-17T15:49:29Z | 2019-10-17T15:49:24Z | MEMBER | dogsheep/twitter-to-sqlite/pulls/24 | Closes #12 and #23 | 206156866 | pull | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/24/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
508578780 | MDU6SXNzdWU1MDg1Nzg3ODA= | 25 | Ensure migrations don't accidentally create foreign key twice | 9599 | closed | 0 | 2 | 2019-10-17T16:08:50Z | 2019-10-17T16:56:47Z | 2019-10-17T16:56:47Z | MEMBER | Is it possible for these lines to run against a database table that already has these foreign keys? https://github.com/dogsheep/twitter-to-sqlite/blob/c9295233f219c446fa2085cace987067488a31b9/twitter_to_sqlite/migrations.py#L21-L22 | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
513074501 | MDU6SXNzdWU1MTMwNzQ1MDE= | 26 | Command for importing mentions timeline | 9599 | closed | 0 | 1 | 2019-10-28T03:14:27Z | 2019-10-30T02:36:13Z | 2019-10-30T02:20:47Z | MEMBER | https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-mentions_timeline Almost identical to home-timeline #18 but it uses `https://api.twitter.com/1.1/statuses/mentions_timeline.json` instead. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
514459062 | MDU6SXNzdWU1MTQ0NTkwNjI= | 27 | retweets-of-me command | 9599 | closed | 0 | 4 | 2019-10-30T07:43:01Z | 2019-11-03T01:12:58Z | 2019-11-03T01:12:58Z | MEMBER | https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets_of_me | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
515658861 | MDU6SXNzdWU1MTU2NTg4NjE= | 28 | Add indexes to followers table | 9599 | closed | 0 | 1 | 2019-10-31T18:40:22Z | 2019-11-09T20:15:42Z | 2019-11-09T20:11:48Z | MEMBER | `select follower_id from following where followed_id = 12497` takes over a second for me at the moment. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/28/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
488833975 | MDU6SXNzdWU0ODg4MzM5NzU= | 3 | Command for running a search and saving tweets for that search | 9599 | closed | 0 | 6 | 2019-09-03T21:29:56Z | 2019-11-04T05:31:56Z | 2019-11-04T05:31:16Z | MEMBER | $ twitter-to-sqlite search dogsheep | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
520508502 | MDU6SXNzdWU1MjA1MDg1MDI= | 31 | "friends" command (similar to "followers") | 9599 | closed | 0 | 2 | 2019-11-09T20:20:20Z | 2022-09-20T05:05:03Z | 2020-02-07T07:03:28Z | MEMBER | Current list of commands: ``` followers Save followers for specified user (defaults to... followers-ids Populate followers table with IDs of account followers friends-ids Populate followers table with IDs of account friends ``` Obvious omission here is `friends`, which would be powered by `https://api.twitter.com/1.1/friends/list.json`: https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-friends-list | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
561454071 | MDU6SXNzdWU1NjE0NTQwNzE= | 32 | Documentation for " favorites" command | 9599 | closed | 0 | 0 | 2020-02-07T06:50:11Z | 2020-02-07T06:59:10Z | 2020-02-07T06:59:10Z | MEMBER | It looks like I forgot to document this one in the README. https://github.com/dogsheep/twitter-to-sqlite/blob/6ebd482619bd94180e54bb7b56549c413077d329/twitter_to_sqlite/cli.py#L183-L194 | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/32/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
561469252 | MDExOlB1bGxSZXF1ZXN0MzcyMjczNjA4 | 33 | Upgrade to sqlite-utils 2.2.1 | 9599 | closed | 0 | 1 | 2020-02-07T07:32:12Z | 2020-03-20T19:21:42Z | 2020-03-20T19:21:41Z | MEMBER | dogsheep/twitter-to-sqlite/pulls/33 | 206156866 | pull | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
585266763 | MDU6SXNzdWU1ODUyNjY3NjM= | 34 | IndexError running user-timeline command | 9599 | closed | 0 | 2 | 2020-03-20T18:54:08Z | 2020-03-20T19:20:52Z | 2020-03-20T19:20:37Z | MEMBER | ``` $ twitter-to-sqlite user-timeline data.db --screen_name Allen_Joines Traceback (most recent call last): File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite", line 11, in <module> load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')() File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 256, in user_timeline utils.save_tweets(db, chunk) File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 289, in save_tweets db["users"].upsert(user, pk="id", alter=True) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py", line 1128, in upsert conversions=conversions, File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py", line 1157, in upsert_all upsert=True, File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py", line 1096, in insert_all row = lis… | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
585282212 | MDU6SXNzdWU1ODUyODIyMTI= | 35 | twitter-to-sqlite user-timeline [screen_names] --sql / --attach | 9599 | closed | 0 | 5 | 2020-03-20T19:26:07Z | 2020-03-20T20:17:00Z | 2020-03-20T20:16:35Z | MEMBER | Split from #8. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
585306847 | MDU6SXNzdWU1ODUzMDY4NDc= | 36 | twitter-to-sqlite followers/friends --sql / --attach | 9599 | closed | 0 | 0 | 2020-03-20T20:20:33Z | 2020-03-20T23:12:38Z | 2020-03-20T23:12:38Z | MEMBER | Split from #8. The `friends` and `followers` commands don't yet support `--sql` and `--attach`. (`friends-ids` and `followers-ids` do though). | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/36/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
585353598 | MDU6SXNzdWU1ODUzNTM1OTg= | 37 | Handle "User not found" error | 9599 | closed | 0 | 3 | 2020-03-20T22:14:32Z | 2020-04-17T23:43:46Z | 2020-04-17T23:43:46Z | MEMBER | While running `user-timeline` I got this bug (because a screen name I asked for didn't exist): ``` File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 185, in transform_user user["created_at"] = parser.parse(user["created_at"]) KeyError: 'created_at' >>> import pdb >>> pdb.pm() > /Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py(185)transform_user() -> user["created_at"] = parser.parse(user["created_at"]) (Pdb) user {'errors': [{'code': 50, 'message': 'User not found.'}]} ``` | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
585359363 | MDU6SXNzdWU1ODUzNTkzNjM= | 38 | Screen name display for user-timeline is uneven | 9599 | closed | 0 | 1 | 2020-03-20T22:30:23Z | 2020-03-20T22:37:17Z | 2020-03-20T22:37:17Z | MEMBER | ``` CDPHE [####################################] 67 CHFSKy [####################################] 3216 DHSWI [####################################] 41 DPHHSMT [####################################] 742 Delaware_DHSS [####################################] 3231 DhhsNevada [####################################] 639 ``` I could format them to match the length of the longest screen name instead. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
590666760 | MDU6SXNzdWU1OTA2NjY3NjA= | 39 | --since feature can be confused by retweets | 9599 | closed | 0 | 11 | 2020-03-30T23:25:33Z | 2020-04-01T03:45:16Z | 2020-04-01T03:45:16Z | MEMBER | If you run `twitter-to-sqlite user-timeline ... --since` it's supposed to fetch Tweets those specific users tweeted since last time the command was run. It does this by seeking out the max ID of their previous tweets: https://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L305-L311 BUT... this has a nasty flaw: if another account had retweeted one of their recent tweets the retweeted-tweet will have been loaded into the database - so we may treat that as the most recent since ID and miss a bunch of their tweets! | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
488835586 | MDU6SXNzdWU0ODg4MzU1ODY= | 4 | Command for importing data from a Twitter Export file | 9599 | closed | 0 | 2 | 2019-09-03T21:34:13Z | 2019-10-11T06:45:02Z | 2019-10-11T06:45:02Z | MEMBER | Twitter lets you export all of your data as an archive file: https://twitter.com/settings/your_twitter_data A command for importing this data into SQLite would be extremely useful. $ twitter-to-sqlite import twitter.db path-to-archive.zip | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
590669793 | MDU6SXNzdWU1OTA2Njk3OTM= | 40 | Feature: record history of follower counts | 9599 | closed | 0 | 5 | 2020-03-30T23:32:28Z | 2020-04-01T04:13:05Z | 2020-04-01T04:13:05Z | MEMBER | We currently over-write the follower count every time we import a tweet (when we import that user profile again): https://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/utils.py#L293-L294 It would be neat if we noticed if that user's follower count (and maybe other counts?) had changed since we last saved them and recorded that change in a separate history table. This would be an inexpensive way of building up rough charts of follower count over time. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
591613579 | MDU6SXNzdWU1OTE2MTM1Nzk= | 41 | Bug: recorded a since_id for None, None | 9599 | closed | 0 | 0 | 2020-04-01T04:29:43Z | 2020-04-01T04:31:11Z | 2020-04-01T04:31:11Z | MEMBER | This shouldn't happen in the `since_ids` table (relates to #39): <img width="422" alt="twitter__since_ids__2_rows" src="https://user-images.githubusercontent.com/9599/78099419-b0fed200-7396-11ea-87d8-992a171d6fab.png"> | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/41/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
602173589 | MDU6SXNzdWU2MDIxNzM1ODk= | 42 | Error running user-timeline with --sql and --ids together | 9599 | closed | 0 | 0 | 2020-04-17T19:02:06Z | 2020-04-17T23:34:40Z | 2020-04-17T23:34:40Z | MEMBER | ``` $ twitter-to-sqlite user-timeline tweets.db --sql='select id from users' --ids Traceback (most recent call last): File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite", line 11, in <module> load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')() File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 284, in user_timeline "@{:" + str(max(len(identifier) for identifier in identifiers)) + "}" File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 284, in <genexpr> "@{:" + str(max(len(identifier) for identifier in identifiers)) + "}" TypeError: object of type 'int' has no len() ``` But this DID work - casting to strings: ``` $ twitter-to-sqlite user-timeline tweets.db --sql='select "" || id from users' --ids ... this worked ... ``` | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/42/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
602176870 | MDU6SXNzdWU2MDIxNzY4NzA= | 43 | "twitter-to-sqlite lists" command for retrieving a user's owned lists | 9599 | closed | 0 | 1 | 2020-04-17T19:08:59Z | 2020-04-17T23:48:28Z | 2020-04-17T23:30:39Z | MEMBER | https://developer.twitter.com/en/docs/accounts-and-users/create-manage-lists/api-reference/get-lists-ownerships `https://api.twitter.com/1.1/lists/ownerships.json ` | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
602181581 | MDU6SXNzdWU2MDIxODE1ODE= | 44 | tweet["source"] can be an empty string | 9599 | closed | 0 | 0 | 2020-04-17T19:18:26Z | 2020-04-17T22:01:44Z | 2020-04-17T22:01:44Z | MEMBER | Got this excepion: ``` File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 641, in extract_and_save_source details = m.groupdict() AttributeError: 'NoneType' object has no attribute 'groupdict' ``` I traced it back to this tweet: https://twitter.com/osder/status/578712651393576960 ``` (Pdb) source_re re.compile('<a href="(?P<url>.*?)".*?>(?P<name>.*?)</a>') (Pdb) locals()['source'] '' (Pdb) u > /Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py(393)save_tweets() -> tweet["source"] = extract_and_save_source(db, tweet["source"]) (Pdb) tweet {'created_at': '2015-03-20T00:20:22+00:00', 'id': 578712651393576960, 'full_text': '@osder', 'truncated': False, 'display_text_range': [0, 6], 'source': '', 'in_reply_to_status_id': 578712521382715392, 'in_reply_to_user_id': 1545741, 'in_reply_to_screen_name': 'osder', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 0, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'lang': 'und', 'user': 1545741} ``` | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/44/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
610284471 | MDU6SXNzdWU2MTAyODQ0NzE= | 46 | Error running 'search' for the first time | 9599 | closed | 0 | 0 | 2020-04-30T18:11:20Z | 2020-04-30T18:11:58Z | 2020-04-30T18:11:58Z | MEMBER | ``` % twitter-to-sqlite search infodemic.db '#infodemic' Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/bin/twitter-to-sqlite", line 11, in <module> load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')() File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/share/virtualenvs/twitter-to-sqlite-PBRUqIv6/lib/python3.7/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/Users/simon/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 867, in search for tweet in tweets: File "/Users/simon/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 165, in fetch_timeline [since_type_id, since_key], sqlite3.OperationalError: no such table: since_ids ``` | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/46/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
663976976 | MDU6SXNzdWU2NjM5NzY5NzY= | 48 | Add a table of contents to the README | 9599 | closed | 0 | 3 | 2020-07-22T18:54:33Z | 2020-07-23T17:46:07Z | 2020-07-22T19:03:02Z | MEMBER | Using https://github.com/jonschlinkert/markdown-toc | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
489419782 | MDU6SXNzdWU0ODk0MTk3ODI= | 6 | Extract extended_entities into a media table | 9599 | closed | 0 | 0 | 2019-09-04T21:59:10Z | 2019-09-04T22:08:01Z | 2019-09-04T22:08:01Z | MEMBER | <img width="747" alt="cleo__select___from_tweets_order_by_id_limit_101" src="https://user-images.githubusercontent.com/9599/64294799-880f7f00-cf24-11e9-83b8-75c1ee54182c.png"> | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
490798130 | MDU6SXNzdWU0OTA3OTgxMzA= | 7 | users-lookup command for fetching users | 9599 | closed | 0 | 0 | 2019-09-08T19:47:59Z | 2019-09-08T20:32:13Z | 2019-09-08T20:32:13Z | MEMBER | https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-users-lookup ``` https://api.twitter.com/1.1/users/lookup.json?user_id=783214,6253282 https://api.twitter.com/1.1/users/lookup.json?screen_name=simonw,cleopaws ``` CLI design: ``` $ twitter-to-sqlite users-lookup simonw cleopaws $ twitter-to-sqlite users-lookup 783214 6253282 --ids ``` | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
490803176 | MDU6SXNzdWU0OTA4MDMxNzY= | 8 | --sql and --attach options for feeding commands from SQL queries | 9599 | closed | 0 | 4 | 2019-09-08T20:35:49Z | 2020-03-20T23:13:01Z | 2020-03-20T23:13:01Z | MEMBER | Say you want to fetch Twitter profiles for a list of accounts that are stored in another database: $ twitter-to-sqlite users-lookup users.db --attach attending.db \ --sql "select Twitter from attending.attendes where Twitter is not null" The SQL query you feed in is expected to return a list of screen names suitable for processing further by the command. Should be supported by all three of: - [x] `twitter-to-sqlite users-lookup` - [x] `twitter-to-sqlite user-timeline` - [x] `twitter-to-sqlite followers` and `friends` The `--attach` option allows other SQLite databases to be attached to the connection. Without it the SQL query will have to read from the single attached database. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
491791152 | MDU6SXNzdWU0OTE3OTExNTI= | 9 | followers-ids and friends-ids subcommands | 9599 | closed | 0 | 1 | 2019-09-10T16:58:15Z | 2019-09-10T17:36:55Z | 2019-09-10T17:36:55Z | MEMBER | These will import follower and friendship IDs into the following tables, using these APIs: https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-ids https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-friends-ids | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
267513424 | MDU6SXNzdWUyNjc1MTM0MjQ= | 1 | Addressable pages for every row in a table | 9599 | closed | 0 | 2857392 | 6 | 2017-10-23T00:44:16Z | 2017-10-24T14:11:04Z | 2017-10-24T14:11:03Z | OWNER | /database-name-7sha256/table-name/compound-pk /database-name-7sha256/table-name/compound-pk.json Tricky part will be figuring out what the private key is - especially since it could be a compound primary key and it might involve different data types. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
267517381 | MDU6SXNzdWUyNjc1MTczODE= | 10 | Set up Travis | 9599 | closed | 0 | 2859414 | 1 | 2017-10-23T01:29:07Z | 2017-11-04T23:48:57Z | 2017-11-04T23:48:57Z | OWNER | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
717746043 | MDExOlB1bGxSZXF1ZXN0NTAwMjU2NDg1 | 1000 | datasette.client internal requests mechanism | 9599 | closed | 0 | 9599 | 5971510 | 18 | 2020-10-08T23:58:25Z | 2020-10-09T16:11:26Z | 2020-10-09T16:11:25Z | OWNER | simonw/datasette/pulls/1000 | Refs #943 | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/1000/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||
717768441 | MDU6SXNzdWU3MTc3Njg0NDE= | 1001 | OPTIONS requests return a 500 error | 9599 | closed | 0 | 5971510 | 8 | 2020-10-09T00:57:13Z | 2020-10-09T01:44:41Z | 2020-10-09T01:43:58Z | OWNER | ``` % curl -vv -XOPTIONS https://latest.datasette.io/ * Trying 216.58.195.83:443... > OPTIONS / HTTP/1.1 > Host: latest.datasette.io > User-Agent: curl/7.70.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 500 Internal Server Error ``` | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1001/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
717783692 | MDU6SXNzdWU3MTc3ODM2OTI= | 1002 | Release notes for Datasette 0.50 | 9599 | closed | 0 | 5971510 | 1 | 2020-10-09T01:45:00Z | 2020-10-09T17:52:54Z | 2020-10-09T17:52:53Z | OWNER | https://github.com/simonw/datasette/compare/0.49.1...c12b7a5def7028845a54a9fdac4052a87a0a8bb8 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1002/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
718255803 | MDU6SXNzdWU3MTgyNTU4MDM= | 1004 | Replace MockRequest with Request.fake() | 9599 | closed | 0 | 5971510 | 0 | 2020-10-09T15:55:28Z | 2020-10-09T16:26:24Z | 2020-10-09T16:26:24Z | OWNER | This code: https://github.com/simonw/datasette/blob/7249ac5ca04b5ddc6517750326ee7e522cc49145/tests/utils.py#L1-L8 Predates the introduction of this class method: https://github.com/simonw/datasette/blob/7249ac5ca04b5ddc6517750326ee7e522cc49145/datasette/utils/asgi.py#L108-L121 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1004/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
718259202 | MDU6SXNzdWU3MTgyNTkyMDI= | 1005 | Remove xfail tests when new httpx is released | 9599 | closed | 0 | 3268330 | 3 | 2020-10-09T16:00:19Z | 2021-02-28T22:41:08Z | 2021-02-28T22:41:08Z | OWNER | > My `httpx` pull request adding `raw_path` support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet. > > I'm going to mark these tests as `xfail` so I can land this change - I'll remove that once an `httpx` release comes out that I can use to get the tests passing. > _Originally posted by @simonw in https://github.com/simonw/datasette/pull/1000#issuecomment-706263157_ | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1005/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
718264811 | MDU6SXNzdWU3MTgyNjQ4MTE= | 1006 | Documentation for datasette.client | 9599 | closed | 0 | 5971510 | 2 | 2020-10-09T16:09:02Z | 2020-10-09T17:22:31Z | 2020-10-09T17:20:37Z | OWNER | > I'm going to document this in a separate issue. _Originally posted by @simonw in https://github.com/simonw/datasette/pull/1000#issuecomment-706269271_ | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1006/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
718484082 | MDU6SXNzdWU3MTg0ODQwODI= | 1010 | json / CSV links are broken in Datasette 0.50 | 9599 | closed | 0 | 3 | 2020-10-10T00:07:42Z | 2020-10-10T02:32:03Z | 2020-10-10T02:32:03Z | OWNER | e.g. on https://latest.datasette.io/fixtures/sortable That export link block is broken. The HTML is: ```html <p class="export-links"> This data as <a href="//fixtures/sortable.json">json</a>, <a href="//fixtures/sortable.csv?_size=max">CSV</a> (<a href="#export">advanced</a>) </p> ``` | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1010/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
718723543 | MDU6SXNzdWU3MTg3MjM1NDM= | 1014 | Add Link: pagination HTTP headers | 9599 | closed | 0 | 6026070 | 6 | 2020-10-10T23:42:40Z | 2020-10-23T19:44:05Z | 2020-10-11T00:18:51Z | OWNER | Spun off from #782. These can go on all of the JSON endpoints that support pagination. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1014/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
718953669 | MDU6SXNzdWU3MTg5NTM2Njk= | 1016 | Add a "delete" icon next to filters (in addition to "remove filter") | 9599 | closed | 0 | 6026070 | 3 | 2020-10-11T23:49:53Z | 2020-10-23T19:44:06Z | 2020-10-12T03:01:58Z | OWNER | The "remove filter" option in the select box is not very discoverable. <img width="549" alt="evernote__notes__85_rows_where_where_source-url_is_not_blank_sorted_by_id" src="https://user-images.githubusercontent.com/9599/95693532-b8265880-0be1-11eb-971e-038dede14cdb.png"> It would be good to have an additional remove icon, pointed to by the pink arrow, which removes a specific selected filter. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1016/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed |