github
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
644283211 | MDU6SXNzdWU2NDQyODMyMTE= | 863 | {{ csrftoken() }} doesn't work with datasette.render_template() | 9599 | closed | 0 | 5533512 | 0 | 2020-06-24T03:11:49Z | 2020-06-24T04:30:30Z | 2020-06-24T03:24:01Z | OWNER | The documentation here suggests that it will work: https://github.com/simonw/datasette/blob/eed116ac0599c7d21b7129af94d58ce03a923e4e/docs/internals.rst#L540-L546 But right now the `csrftoken` variable is set in BaseView.render, which means it's not visible to plugins that try to render templates using `datasette.render_template`: https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/base.py#L99-L106 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/863/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
637409144 | MDU6SXNzdWU2Mzc0MDkxNDQ= | 839 | {"$file": ...} mechanism is broken | 9599 | closed | 0 | 5512395 | 0 | 2020-06-12T00:46:24Z | 2020-06-12T00:48:26Z | 2020-06-12T00:48:26Z | OWNER | https://travis-ci.org/github/simonw/datasette/jobs/697445318 ``` def test_plugin_config_file(app_client): open(TEMP_PLUGIN_SECRET_FILE, "w").write("FROM_FILE") > assert {"foo": "FROM_FILE"} == app_client.ds.plugin_config("file-plugin") E AssertionError: assert {'foo': 'FROM_FILE'} == {'foo': {'$fi...ugin-secret'}} E Differing items: E {'foo': 'FROM_FILE'} != {'foo': {'$file': '/tmp/plugin-secret'}} E Use -v to get the full diff ``` Broken in https://github.com/simonw/datasette/commit/fba8ff6e76253af2b03749ed8dd6e28985a7fb8f as part of #837 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/839/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
978743426 | MDU6SXNzdWU5Nzg3NDM0MjY= | 13 | xml.etree.ElementTree.ParseError: not well-formed (invalid token) | 9599 | closed | 0 | 4 | 2021-08-25T05:48:21Z | 2021-08-26T18:45:13Z | 2021-08-26T18:45:13Z | MEMBER | Got this error today: ``` (evernote-to-sqlite) /tmp % evernote-to-sqlite enex evernote.db simonwillison\'s\ notebook.enex Importing from ENEX [######------------------------------] 17% Traceback (most recent call last): File "/Users/simon/.local/bin/evernote-to-sqlite", line 8, in <module> sys.exit(cli()) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 763, in invoke return __callback(*args, **kwargs) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/evernote_to_sqlite/cli.py", line 31, in enex save_note(db, note) File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/evernote_to_sqlite/utils.py", line 36, in save_note content = ET.tostring(ET.fromstring(content_xml)).decode("utf-8") File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py", line 1347, in XML parser.feed(text) xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 132 ``` | 303218369 | issue | { "url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
986829194 | MDU6SXNzdWU5ODY4MjkxOTQ= | 14 | xml.etree.ElementTree.Parse Error - mismatched tag | 46968 | open | 0 | 1 | 2021-09-02T14:46:36Z | 2021-09-02T14:53:11Z | NONE | This is an error message I get upon parsing the enex file of my Inbox. Please find the full error message below. Any hints welcome. ``` Importing from ENEX [##################------------------] 50% 00:00:50 Traceback (most recent call last): File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/bin/evernote-to-sqlite", line 8, in <module> sys.exit(cli()) File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py", line 763, in invoke return __callback(*args, **kwargs) File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/cli.py", line 30, in enex for tag, note in find_all_tags(fp, ["note"], progress_callback=bar.update): File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/utils.py", line 17, in find_all_tags for event, el in parser.read_events(): File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py", line 1329, in read_events raise event File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py", line 1301, in feed self._parser.feed(data) xml.etree.ElementTree.ParseError: mismatc… | 303218369 | issue | { "url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/14/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1393903845 | I_kwDOBm6k_c5TFUjl | 1828 | word-wrap: anywhere resulting in weird display | 9599 | closed | 0 | 2 | 2022-10-02T21:25:03Z | 2022-10-02T23:01:17Z | 2022-10-02T23:01:17Z | OWNER | e.g. on https://github-to-sqlite.dogsheep.net/github/commits <img width="893" alt="image" src="https://user-images.githubusercontent.com/9599/193476890-a96fae4f-4883-4698-816d-90f9cf6efd6c.png"> This is from a change introduced here: https://github.com/simonw/datasette/commit/bf8d84af5422606597be893cedd375020cb2b369 in #1805 https://github.com/simonw/datasette/blob/bf8d84af5422606597be893cedd375020cb2b369/datasette/static/app.css#L447-L450 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1828/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1089529555 | I_kwDOBm6k_c5A8ObT | 1581 | when hashed urls are turned on, the _memory db has improperly long-lived cache expiry | 536941 | closed | 0 | 1 | 2021-12-28T00:05:48Z | 2022-03-24T04:08:18Z | 2022-03-24T04:08:18Z | CONTRIBUTOR | if hashed_urls are on, then a -000 suffix is added to the `_memory` database, and the cache settings are set just as if it was a normal hashed database. in particular, this header is set: `cache-control: max-age=31536000` this is not appropriate because the `_memory-000` database isn't really hashed based on the contents of the databases (see #1561). Either the cache-control header should be changed, or the _memory db should have a hash suffix that does depend on the contents of the databases. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1581/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
789336592 | MDU6SXNzdWU3ODkzMzY1OTI= | 1195 | view_name = "query" for the query page | 9599 | open | 0 | 4 | 2021-01-19T20:21:36Z | 2021-01-25T04:40:08Z | OWNER | It uses `view_name` of `database` at the moment which isn't as useful. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1195/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1076057610 | I_kwDOBm6k_c5AI1YK | 1546 | validating the sql | 50336793 | closed | 0 | 1 | 2021-12-09T21:35:57Z | 2021-12-18T02:05:17Z | 2021-12-18T02:05:16Z | NONE | Could someone tell me that part of the code is responsible for validating the sql that guarantees that only a table can be read | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1546/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
795367402 | MDU6SXNzdWU3OTUzNjc0MDI= | 1209 | v0.54 500 error from sql query in custom template; code worked in v0.53; found a workaround | 11788561 | open | 0 | 1 | 2021-01-27T19:08:13Z | 2021-01-28T23:00:27Z | NONE | v0.54 500 error in sql query template; code worked in v0.53; found a workaround **schema:** CREATE TABLE "talks" ("talk" TEXT,"series" INTEGER, "talkdate" TEXT) CREATE TABLE "series" ("id" INTEGER PRIMARY KEY, "series" TEXT, talks_list TEXT default '', website TEXT default ''); **Live example of correctly rendered template in v.053:** https://cosmotalks-cy6xkkbezq-uw.a.run.app/cosmotalks/talks/1 **Description of problem:** I needed 'sql select' code in a custom row-mydatabase-mytable.html template to lookup the series name for a foreign key integer value in the talks table. So `metadata.json` specifies the `datasette-template-sql` plugin. The code below worked perfectly in v0.53 (just the relevant sql statement part is shown; full code is [here](https://github.com/jrdmb/cosmotalks-datasette/blob/main/templates/row-cosmotalks-talks.html)): ``` {# custom addition #} {% for row in display_rows %} ... {% set sname = sql("select series from series where id = ?", [row.series]) %} <strong>Series name: {{ sname[0].series }} ... {% endfor %} {# End of custom addition #} ``` **In v0.54, that code resulted in a 500 error with a 'no such table series' message.** A second query in that template also did not work but the above is fully illustrative of the problem. All templates were up-to-date along with datasette v0.54. **Workaround:** After fiddling around with trying different things, what worked was the syntax from [Querying a different database from the datasette-template-sql github repo](https://github.com/simonw/datasette-template-sql#querying-a-different-database) to add the database name to the sql statement: `{% set sname = sql("select series from series where id = ?", [row.series], database="cosmotalks") %}` Though this was found to work, it should not be necessary to add `database="cosmotalks"` since per the `datasette-template-sql` README, it's only needed when querying a different database, but here it's a table within the same databa… | 107914493 | issue | |||||||||
955316250 | MDU6SXNzdWU5NTUzMTYyNTA= | 1405 | utils.parse_metadata() should be a documented internal function | 9599 | closed | 0 | 3 | 2021-07-28T23:51:39Z | 2021-07-29T23:33:30Z | 2021-07-29T23:30:24Z | OWNER | Because it's used by this plugin: https://github.com/simonw/datasette-remote-metadata | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1405/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
490798130 | MDU6SXNzdWU0OTA3OTgxMzA= | 7 | users-lookup command for fetching users | 9599 | closed | 0 | 0 | 2019-09-08T19:47:59Z | 2019-09-08T20:32:13Z | 2019-09-08T20:32:13Z | MEMBER | https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-users-lookup ``` https://api.twitter.com/1.1/users/lookup.json?user_id=783214,6253282 https://api.twitter.com/1.1/users/lookup.json?screen_name=simonw,cleopaws ``` CLI design: ``` $ twitter-to-sqlite users-lookup simonw cleopaws $ twitter-to-sqlite users-lookup 783214 6253282 --ids ``` | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1650984552 | PR_kwDOJHON9s5NbyYN | 13 | use universal command | 14314871 | open | 0 | 0 | 2023-04-02T15:10:54Z | 2023-04-02T15:37:34Z | FIRST_TIME_CONTRIBUTOR | dogsheep/apple-notes-to-sqlite/pulls/13 | 611552758 | pull | { "url": "https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||||
1556065335 | PR_kwDOBm6k_c5Ie5nA | 2004 | use single quotes for string literals, fixes #2001 | 193185 | open | 0 | 1 | 2023-01-25T05:08:46Z | 2023-02-01T06:37:18Z | CONTRIBUTOR | simonw/datasette/pulls/2004 | This modernizes some uses of double quotes for string literals to use only single quotes, fixes simonw/datasette#2001 While developing it, I manually enabled the stricter mode by using the code snippet at https://gist.github.com/cldellow/85bba507c314b127f85563869cd94820 I think that code snippet isn't generally safe/portable, so I haven't tried to automate it in the tests. <!-- readthedocs-preview datasette start --> ---- :books: Documentation preview :books:: https://datasette--2004.org.readthedocs.build/en/2004/ <!-- readthedocs-preview datasette end --> | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/2004/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
1884499674 | PR_kwDODFE5qs5ZtYMc | 13 | use poetry for packages, asdf for versioning, and gh actions for ci | 150855 | open | 0 | 0 | 2023-09-06T17:59:16Z | 2023-09-06T17:59:16Z | FIRST_TIME_CONTRIBUTOR | dogsheep/google-takeout-to-sqlite/pulls/13 | - build: use poetry for package management, asdf for python version - build: cleanup poetry config, add keywords, ignore dist - ci: migrate circleci to gh actions - fix: dup method definition | 206649770 | pull | { "url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
752888228 | MDExOlB1bGxSZXF1ZXN0NTI5MDkwNTYw | 204 | use jsonify_if_need for sql updates | 78035 | closed | 0 | 1 | 2020-11-29T10:49:00Z | 2020-12-08T17:49:42Z | 2020-12-08T17:49:42Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/204 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/204/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
1400121355 | PR_kwDOBm6k_c5AVujU | 1835 | use inspect data for hash and file size | 536941 | closed | 0 | 3 | 2022-10-06T18:25:24Z | 2022-10-27T20:51:30Z | 2022-10-06T20:06:07Z | CONTRIBUTOR | simonw/datasette/pulls/1835 | `inspect_data` should already include the hash and the db file size, so this PR takes advantage of using those instead of always recalculating. should help a lot on startup with large DBs. closes #1834 | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/1835/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
545407916 | MDU6SXNzdWU1NDU0MDc5MTY= | 73 | upsert_all() throws issue when upserting to empty table | 82988 | closed | 0 | 6 | 2020-01-05T11:58:57Z | 2020-01-31T14:21:09Z | 2020-01-05T17:20:18Z | NONE | If I try to add a list of `dict`s to an empty table using `upsert_all`, I get an error: ```python import sqlite3 from sqlite_utils import Database import pandas as pd conx = sqlite3.connect(':memory') cx = conx.cursor() cx.executescript('CREATE TABLE "test" ("Col1" TEXT);') q="SELECT * FROM test;" pd.read_sql(q, conx) #shows empty table db = Database(conx) db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}]) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-74-8c26d93d7587> in <module> 1 db = Database(conx) ----> 2 db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}]) /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts) 1157 alter=alter, 1158 extracts=extracts, -> 1159 upsert=True, 1160 ) 1161 /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert) 1040 sql = "INSERT OR IGNORE INTO [{table}]({pks}) VALUES({pk_placeholders});".format( 1041 table=self.name, -> 1042 pks=", ".join(["[{}]".format(p) for p in pks]), 1043 pk_placeholders=", ".join(["?" for p in pks]), 1044 ) TypeError: 'NoneType' object is not iterable ``` A hacky workaround in use is: ```python try: db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}]) except: db['test'].insert_all([{'Col1':'a'},{'Col1':'b'}]) ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/73/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1465194249 | I_kwDOCGYnMM5XVRcJ | 514 | upsert of new row with check constraints fails | 193185 | closed | 0 | 5 | 2022-11-26T16:12:23Z | 2023-05-08T21:50:52Z | 2023-05-08T21:50:51Z | NONE | (I originally opened this in https://github.com/simonw/datasette-insert/issues/20, but I see that that library depends on sqlite-utils) In the case of a new row, upsert first adds the row, specifying only its pkeys: https://github.com/simonw/sqlite-utils/blob/965ca0d5f5bffe06cc02cd7741344d1ddddf9d56/sqlite_utils/db.py#L2783-L2787 This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/514/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1465194930 | PR_kwDOCGYnMM5DvZxa | 515 | upsert new rows with constraints, fixes #514 | 193185 | closed | 0 | 1 | 2022-11-26T16:15:21Z | 2023-05-08T21:27:11Z | 2023-05-08T21:27:10Z | NONE | simonw/sqlite-utils/pulls/515 | This fixes #514 by making the initial insert for upserts include all columns, so that new rows can be added to tables with non-pkey columns that have constraints. (aside: I'm not a python programmer. `pip`? `pipenv`? `venv`? These are mystical incantations to me. The process to set up this repo for local development and testing was _so easy_. Thank you for the excellent contributing documentation!) <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--515.org.readthedocs.build/en/515/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/515/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1094974713 | I_kwDOCGYnMM5BQ_z5 | 362 | upsert --detect-types is broken | 9599 | closed | 0 | 0 | 2022-01-06T05:12:10Z | 2022-01-06T06:54:45Z | 2022-01-06T06:28:34Z | OWNER | Noticed this thanks to syntax highlighting in VS Code showing an unused variable - need to fix it and add a test. <img src="https://user-images.githubusercontent.com/9599/148331804-6295bcf5-01fd-4159-b8ad-60c82acb97b4.png" width="200"> | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/362/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
605938063 | MDU6SXNzdWU2MDU5MzgwNjM= | 9 | upload command should be resumable, should only upload photos not already uploaded | 9599 | closed | 0 | 2 | 2020-04-23T23:31:08Z | 2020-04-23T23:39:14Z | 2020-04-23T23:39:14Z | MEMBER | Follow on from #4. | 256834907 | issue | { "url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
506297048 | MDU6SXNzdWU1MDYyOTcwNDg= | 594 | upgrade to uvicorn-0.9 to be Python-3.8 friendly | 4312421 | closed | 0 | 3 | 2019-10-13T09:23:43Z | 2019-11-12T04:47:04Z | 2019-11-12T04:47:04Z | NONE | uvicorn-0.8 relies on websockets-0.7 which lacks python-3.8 compatiblity | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/594/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
527670799 | MDU6SXNzdWU1Mjc2NzA3OTk= | 639 | updating metadata.json without recreating the app | 172847 | open | 0 | 6 | 2019-11-24T09:19:53Z | 2019-11-30T06:08:50Z | NONE | I've sucessfully "uploaded" an SQLite database (with a metadata.json file) to heroku using: $ datasette publish heroku so-sales.db -m metadata.json -n so-sales The question is: how can I modify the (small) metadata.json file without having to upload the (large) SQLite database. The directions on heroku indicate I should run: heroku git:clone -a so-sales But this just results in an empty directory with a warning: warning: You appear to have cloned an empty repository. I've been able to "clone" the heroku "app" using the command: $ heroku slugs:download -a so-sales but this is not a git repository.... Ideally, it seems to me, there'd be an option of the `datasette` CLI to allow a file to be updated, or there'd be some way to create a local git "clone" of the app so that the heroku instructions for "Deploying with git" would apply. (p.s. I ran `datasette publish heroku -m metadata.json -n so-sales` in the hope that that would not cause the .db file to be wiped, but of course it was.) (p.p.s. Thanks for Datasette!) | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/639/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
950664971 | MDU6SXNzdWU5NTA2NjQ5NzE= | 1401 | unordered list is not rendering bullet points in description_html on database page | 536941 | open | 0 | 2 | 2021-07-22T13:24:18Z | 2021-10-23T13:09:10Z | CONTRIBUTOR | Thanks for this tremendous package, @simonw! In the `description_html` for a database, I [have an unordered list](https://github.com/labordata/warehouse/blob/fcea4502e5b615b0eb3e0bdcb45ec634abe20bb6/warehouse_metadata.yml#L19-L22). However, on the database page on the deployed site, it is not rendering this as a bulleted list. ![Screenshot 2021-07-22 at 09-21-51 nlrb](https://user-images.githubusercontent.com/536941/126645923-2777b7f1-fd4c-4d2d-af70-a35e49a07675.png) Page here: https://labordata-warehouse.herokuapp.com/nlrb-9da4ae5 The documentation gives an [example of using an unordered list](https://docs.datasette.io/en/stable/metadata.html#using-yaml-for-metadata) in a `description_html`, so I expected this will work. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1401/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
585282212 | MDU6SXNzdWU1ODUyODIyMTI= | 35 | twitter-to-sqlite user-timeline [screen_names] --sql / --attach | 9599 | closed | 0 | 5 | 2020-03-20T19:26:07Z | 2020-03-20T20:17:00Z | 2020-03-20T20:16:35Z | MEMBER | Split from #8. | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
505666744 | MDExOlB1bGxSZXF1ZXN0MzI3MDUxNjcz | 15 | twitter-to-sqlite import command, refs #4 | 9599 | closed | 0 | 0 | 2019-10-11T06:37:14Z | 2019-10-11T06:45:01Z | 2019-10-11T06:45:01Z | MEMBER | dogsheep/twitter-to-sqlite/pulls/15 | 206156866 | pull | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
585306847 | MDU6SXNzdWU1ODUzMDY4NDc= | 36 | twitter-to-sqlite followers/friends --sql / --attach | 9599 | closed | 0 | 0 | 2020-03-20T20:20:33Z | 2020-03-20T23:12:38Z | 2020-03-20T23:12:38Z | MEMBER | Split from #8. The `friends` and `followers` commands don't yet support `--sql` and `--attach`. (`friends-ids` and `followers-ids` do though). | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/36/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
602181581 | MDU6SXNzdWU2MDIxODE1ODE= | 44 | tweet["source"] can be an empty string | 9599 | closed | 0 | 0 | 2020-04-17T19:18:26Z | 2020-04-17T22:01:44Z | 2020-04-17T22:01:44Z | MEMBER | Got this excepion: ``` File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 641, in extract_and_save_source details = m.groupdict() AttributeError: 'NoneType' object has no attribute 'groupdict' ``` I traced it back to this tweet: https://twitter.com/osder/status/578712651393576960 ``` (Pdb) source_re re.compile('<a href="(?P<url>.*?)".*?>(?P<name>.*?)</a>') (Pdb) locals()['source'] '' (Pdb) u > /Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py(393)save_tweets() -> tweet["source"] = extract_and_save_source(db, tweet["source"]) (Pdb) tweet {'created_at': '2015-03-20T00:20:22+00:00', 'id': 578712651393576960, 'full_text': '@osder', 'truncated': False, 'display_text_range': [0, 6], 'source': '', 'in_reply_to_status_id': 578712521382715392, 'in_reply_to_user_id': 1545741, 'in_reply_to_screen_name': 'osder', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 0, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'lang': 'und', 'user': 1545741} ``` | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/44/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1363552780 | I_kwDOBm6k_c5RRioM | 1805 | truncate_cells_html does not work for links? | 562352 | open | 0 | 7 | 2022-09-06T16:41:29Z | 2022-10-03T09:18:06Z | NONE | We have many links inside our dataset (please don't blame us ;-). When I use `--settings truncate_cells_html 60` it is not working for the links. Eg. https://images.openfoodfacts.org/images/products/000/000/000/088/nutrition_fr.5.200.jpg (87 chars) is not truncated: ![image](https://user-images.githubusercontent.com/562352/188689045-1946d776-2305-47cf-bfc5-b5685b9206b7.png) IMHO It would make sense that links should be treated as HTML. The link should work of course, but Datasette could truncate it: [https://images.openfoodfacts.org/images/products/00[...].jpg](https://images.openfoodfacts.org/images/products/000/000/000/088/nutrition_fr.5.200.jpg) | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1805/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
reopened | |||||||
858501079 | MDU6SXNzdWU4NTg1MDEwNzk= | 255 | transform --help should tell you the available types | 9599 | closed | 0 | 0 | 2021-04-15T05:24:48Z | 2021-05-29T03:55:52Z | 2021-05-29T03:55:52Z | OWNER | ``` Usage: sqlite-utils transform [OPTIONS] PATH TABLE Transform a table beyond the capabilities of ALTER TABLE Options: --type <TEXT TEXT>... Change column type to X ``` This should specify that the possible types are 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/255/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
386459810 | MDExOlB1bGxSZXF1ZXN0MjM1MTk0Mjg2 | 390 | tiny typo in customization docs | 418191 | closed | 0 | 1 | 2018-12-01T13:44:42Z | 2019-12-19T02:30:35Z | 2018-12-16T21:32:56Z | CONTRIBUTOR | simonw/datasette/pulls/390 | was looking to add some custom templates to my use of datasette and saw this small typo. | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/390/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
703970814 | MDU6SXNzdWU3MDM5NzA4MTQ= | 24 | the JSON object must be str, bytes or bytearray, not 'Undefined' | 9599 | closed | 0 | 8 | 2020-09-17T23:21:41Z | 2020-09-18T22:33:32Z | 2020-09-18T22:33:32Z | MEMBER | Got this on a search results page. | 197431109 | issue | { "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1423000702 | I_kwDOCGYnMM5U0UR- | 503 | test_recreate failing on Windows Python 3.11 | 9599 | closed | 0 | 10 | 2022-10-25T20:01:41Z | 2022-10-25T20:47:34Z | 2022-10-25T20:45:43Z | OWNER | https://github.com/simonw/sqlite-utils/actions/runs/3323672128/jobs/5494726927 Related: - #502 ``` FAILED tests/test_recreate.py::test_recreate[True-True] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_recreate_True_True_0\\data.db' FAILED tests/test_recreate.py::test_recreate[False-True] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_recreate_False_True_0\\data.db' ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/503/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
519032008 | MDExOlB1bGxSZXF1ZXN0MzM3ODQ3NTcz | 64 | test_insert_upsert_all_empty_list | 9599 | closed | 0 | 0 | 2019-11-07T04:24:45Z | 2019-11-07T04:32:38Z | 2019-11-07T04:32:38Z | OWNER | simonw/sqlite-utils/pulls/64 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/64/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
1781047747 | I_kwDOBm6k_c5qKKHD | 2092 | test_homepage intermittent failure | 9599 | closed | 0 | 2 | 2023-06-29T15:20:37Z | 2023-06-29T15:26:28Z | 2023-06-29T15:24:13Z | OWNER | e.g. in https://github.com/simonw/datasette/actions/runs/5413590227/jobs/9839373852 ``` =================================== FAILURES =================================== ________________________________ test_homepage _________________________________ [gw0] linux -- Python 3.7.17 /opt/hostedtoolcache/Python/3.7.17/x64/bin/python ds_client = <datasette.app.DatasetteClient object at 0x7f85d271ef50> @pytest.mark.asyncio async def test_homepage(ds_client): response = await ds_client.get("/.json") assert response.status_code == 200 assert "application/json; charset=utf-8" == response.headers["content-type"] data = response.json() assert data.keys() == {"fixtures": 0}.keys() d = data["fixtures"] assert d["name"] == "fixtures" assert d["tables_count"] == 24 assert len(d["tables_and_views_truncated"]) == 5 assert d["tables_and_views_more"] is True # 4 hidden FTS tables + no_primary_key (hidden in metadata) assert d["hidden_tables_count"] == 6 # 201 in no_primary_key, plus 6 in other hidden tables: > assert d["hidden_table_rows_sum"] == 207, data E AssertionError: {'fixtures': {'color': '9403e5', 'hash': None, 'hidden_table_rows_sum': 0, 'hidden_tables_count': 6, ...}} E assert 0 == 207 ``` My guess is that this is a timing error, where very occasionally the "count rows but stop counting if it exceeds a time limit" thing fails. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/2092/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
328171513 | MDU6SXNzdWUzMjgxNzE1MTM= | 302 | test-2.3.sqlite database filename throws a 404 | 9599 | closed | 0 | 3439337 | 2 | 2018-05-31T14:50:58Z | 2018-06-21T15:21:17Z | 2018-06-21T15:21:16Z | OWNER | The following almost works: datasette test-2.3.sqlite http://127.0.0.1:8001test-2.3-c88bc35/HighWays loads OK, but http://127.0.0.1:8001test-2.3-c88bc35 throws a 404: ![2018-05-31 at 7 50 am](https://user-images.githubusercontent.com/9599/40789434-447ae934-64a7-11e8-9a07-4eeba87147d5.png) | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/302/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1402608214 | PR_kwDOBm6k_c5AdyZ4 | 1840 | test commit | 102635518 | closed | 0 | 0 | 2022-10-10T05:15:26Z | 2022-10-10T09:11:50Z | 2022-10-10T09:11:50Z | FIRST_TIMER | simonw/datasette/pulls/1840 | lalalalalalala <!-- readthedocs-preview datasette start --> ---- :books: Documentation preview :books:: https://datasette--1840.org.readthedocs.build/en/1840/ <!-- readthedocs-preview datasette end --> | 107914493 | pull | { "url": "https://api.github.com/repos/simonw/datasette/issues/1840/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
704685890 | MDU6SXNzdWU3MDQ2ODU4OTA= | 25 | template_debug mechanism | 9599 | closed | 0 | 2 | 2020-09-18T22:11:09Z | 2020-09-18T22:12:21Z | 2020-09-18T22:12:03Z | MEMBER | > I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible. _Originally posted by @simonw in https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584_ | 197431109 | issue | { "url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
919702451 | MDU6SXNzdWU5MTk3MDI0NTE= | 271 | table.upsert_all() fails if input has a single column that should be a primary key | 9599 | closed | 0 | 1 | 2021-06-13T02:50:27Z | 2021-06-13T02:57:29Z | 2021-06-13T02:57:29Z | OWNER | This works: ```pycon >>> db['foo'].insert_all([{"name": "hello"}], pk="name") <Table foo (name)> ``` But this fails: ``` >>> db['foo3'].upsert_all([{"name": "hello"}], pk="name") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 1837, in upsert_all return self.insert_all( File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 1778, in insert_all self.insert_chunk( File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 1588, in insert_chunk result = self.db.execute(query, params) File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 213, in execute return self.conn.execute(sql, parameters) sqlite3.OperationalError: near "WHERE": syntax error ``` With the debugger: ``` >>> import pdb; pdb.pm() > /Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py(213)execute() -> return self.conn.execute(sql, parameters) (Pdb) print(sql, parameters) UPDATE [foo3] SET WHERE [name] = ? ['hello'] ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/271/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
462430920 | MDU6SXNzdWU0NjI0MzA5MjA= | 35 | table.update(...) method | 9599 | closed | 0 | 2 | 2019-06-30T18:06:15Z | 2019-07-28T15:43:52Z | 2019-07-28T15:43:52Z | OWNER | Spun off from #23 - this method will allow a user to update a specific row. Currently the only way to do that it is to call `.upsert({full record})` with the primary key field matching an existing record - but this does not support partial updates. ```python db["events"].update(3, {"name": "Renamed"}) ``` This method only works on an existing table, so there's no need for a `pk="id"` specifier - it can detect the primary key by looking at the table. If the primary key is compound the first argument can be a tuple: ```python db["events_venues"].update((3, 2), {"custom_label": "Label"}) ``` The method can be called without the second dictionary argument. Doing this selects the row specified by the primary key (throwing an error if it does not exist) and remembers it so that chained operations can be carried out - see proposal in https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507055345 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/35/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
467862459 | MDExOlB1bGxSZXF1ZXN0Mjk3NDEyNDY0 | 38 | table.update() method | 9599 | closed | 0 | 2 | 2019-07-14T17:03:49Z | 2019-07-28T15:43:51Z | 2019-07-28T15:43:51Z | OWNER | simonw/sqlite-utils/pulls/38 | Refs #35 Still to do: - [x] Unit tests - [x] Switch to using `.get()` - [x] Better exceptions, plus unit tests for what happens if pk does not exist - [x] Documentation - [x] Ensure compound primary keys work properly - [x] `alter=True` support | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/38/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
777386465 | MDU6SXNzdWU3NzczODY0NjU= | 211 | table.triggers_dict introspection property | 9599 | closed | 0 | 0 | 2021-01-02T02:04:00Z | 2021-01-02T02:10:10Z | 2021-01-02T02:10:10Z | OWNER | `table.triggers` currently returns a list of `Trigger` values. A `table.triggers_dict` property could behave like `columns_dict`, returning a dictionary mapping trigger names to their SQL definitions for that table. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/211/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
621989740 | MDU6SXNzdWU2MjE5ODk3NDA= | 114 | table.transform() method for advanced alter table | 9599 | closed | 0 | 5897911 | 26 | 2020-05-20T18:20:46Z | 2020-09-22T07:51:37Z | 2020-09-22T04:20:02Z | OWNER | SQLite's `ALTER TABLE` can only do the following: * Rename a table * Rename a column * Add a column Notably, it cannot drop columns - so tricks like "add a float version of this text column, populate it, then drop the old one and rename" won't work. The docs here https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes describe a way of implementing full alters safely within a transaction, but it's fiddly. 1. Create new table 2. Copy data 3. Drop old table 4. Rename new into old It would be great if `sqlite-utils` provided an abstraction to help make these kinds of changes safely. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
705975133 | MDExOlB1bGxSZXF1ZXN0NDkwNjA3OTQ5 | 161 | table.transform() method | 9599 | closed | 0 | 5897911 | 13 | 2020-09-21T23:16:59Z | 2020-09-22T07:48:24Z | 2020-09-22T04:20:02Z | OWNER | simonw/sqlite-utils/pulls/161 | Refs #114 - [x] Ability to change the primary key - [x] Support for changing default value for columns - [x] Support for changing `NOT NULL` status of columns - [x] Support for copying existing foreign keys and removing them - <strike>Support for `conversions=` parameter</strike> - [x] Detailed documentation - [x] `PRAGMA foreign_keys` stuff | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||
735663855 | MDExOlB1bGxSZXF1ZXN0NTE1MDE0ODgz | 195 | table.search() improvements plus sqlite-utils search command | 9599 | closed | 0 | 3 | 2020-11-03T22:02:08Z | 2020-11-06T18:30:49Z | 2020-11-06T18:30:42Z | OWNER | simonw/sqlite-utils/pulls/195 | Refs #192. Still needs tests. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/195/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
816560819 | MDU6SXNzdWU4MTY1NjA4MTk= | 240 | table.pks_and_rows_where() method returning primary keys along with the rows | 9599 | closed | 0 | 7 | 2021-02-25T15:49:28Z | 2021-02-25T16:39:23Z | 2021-02-25T16:28:23Z | OWNER | *Original title: Easier way to update a row returned from .rows* Here's a surprisingly hard problem I ran into while trying to implement #239 - given a row returned by `db[table].rows` how can you update that row? The problem is that the `db[table].update(...)` method requires a primary key. But if you have a row from the `db[table].rows` iterator it might not even contain the primary key - provided the table is a `rowid` table. Instead, currently, you need to introspect the table and, if `rowid` is a primary key, explicitly include that in the `select=` argument to `table.rows_where(...)` - otherwise it will not be returned. A utility mechanism to make this easier would be very welcome. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/240/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
695377804 | MDU6SXNzdWU2OTUzNzc4MDQ= | 153 | table.optimize() should delete junk rows from *_fts_docsize | 9599 | closed | 0 | 3 | 2020-09-07T20:31:09Z | 2020-09-24T20:35:46Z | 2020-09-07T21:16:33Z | OWNER | > The second challenge here is cleaning up all of those junk rows in existing `*_fts_docsize` tables. Doing that just to the demo database from https://github-to-sqlite.dogsheep.net/github.db dropped its size from 22MB to 16MB! Here's the SQL: > ```sql > DELETE FROM [licenses_fts_docsize] WHERE id NOT IN ( > SELECT rowid FROM [licenses_fts]); > ``` > I can do that as part of the existing `table.optimize()` method, which optimizes FTS tables. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688501064_ | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/153/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
467864071 | MDU6SXNzdWU0Njc4NjQwNzE= | 39 | table.get(...) method | 9599 | closed | 0 | 0 | 2019-07-14T17:20:51Z | 2019-07-15T04:28:53Z | 2019-07-15T04:28:53Z | OWNER | Utility method for fetching a record by its primary key. Accepts a single value (for primary key / rowid tables) or a list/tuple of values (for compound primary keys, refs #36). Raises a `NotFoundError` if the record cannot be found. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/39/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
470345929 | MDU6SXNzdWU0NzAzNDU5Mjk= | 42 | table.extract(...) method and "sqlite-utils extract" command | 9599 | closed | 0 | 5897911 | 21 | 2019-07-19T14:09:36Z | 2020-09-22T23:39:31Z | 2020-09-22T23:37:49Z | OWNER | One of my favourite features of [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite) is that it can "extract" columns into a separate lookup table - for example: csvs-to-sqlite big_csv_file.csv -c country output.db This will turn the `country` column in the resulting table into a integer foreign key against a new `country` table. You can see an example of what that looks like here: https://san-francisco.datasettes.com/registered-business-locations-3d50679/Business+Corridor was extracted from https://san-francisco.datasettes.com/registered-business-locations-3d50679/Registered_Business_Locations_-_San_Francisco?Business%20Corridor=1 I'd like to have the same capability in `sqlite-utils` - but with the ability to run it against an existing SQLite table rather than just against a CSV. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
705190723 | MDU6SXNzdWU3MDUxOTA3MjM= | 160 | table.enable_fts(..., replace=True) | 9599 | closed | 0 | 5896742 | 1 | 2020-09-20T21:36:23Z | 2020-09-24T20:35:47Z | 2020-09-20T22:05:51Z | OWNER | I noticed that https://til.simonwillison.net/ search doesn't use porter stemming. I'd like to add that, but since [the build script](https://github.com/simonw/til/blob/9d3f0fca30e94df3970df52b0447907a077e4673/build_database.py) always operates on an existing database (to avoid re-rendering markdown and re-building image thumbnails) I'd like it to only add porter stemming if it's not there already. So I'd like to be able to say "set up FTS to look like this, and fix it if it doesn't". I think the neatest way to do that is with a `replace=True` argument to `.enable_fts()`, for consistency with `def .create_view(self, name, sql, replace=True)`. So the `replace=True` argument would check and see if the configured FTS exists already with the correct options (columns, stemming, triggers) - and if any of those are incorrect it would call `.disable_fts()` and then create a new FTS configuration with the correct options. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/160/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
571805300 | MDU6SXNzdWU1NzE4MDUzMDA= | 88 | table.disable_fts() method and "sqlite-utils disable-fts ..." command | 9599 | closed | 0 | 5 | 2020-02-27T04:00:50Z | 2020-02-27T04:40:44Z | 2020-02-27T04:40:44Z | OWNER | This would make it easier to iterate on the FTS configuration for a database without having to wipe and recreate the database each time. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/88/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1353189941 | I_kwDOCGYnMM5QqAo1 | 475 | table.default_values introspection property | 9599 | closed | 0 | 8355157 | 1 | 2022-08-27T22:33:31Z | 2022-08-27T22:44:46Z | 2022-08-27T22:43:02Z | OWNER | > Interesting challenge with `default_value`: I need to be able to tell if the default values passed to `.create()` differ from those in the database already. > > Introspecting that is a bit tricky: > > ```pycon > >>> import sqlite_utils > >>> db = sqlite_utils.Database(memory=True) > >>> db["blah"].create({"id": int, "name": str}, not_null=("name",), defaults={"name": "bob"}) > <Table blah (id, name)> > >>> db["blah"].columns > [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value="'bob'", is_pk=0)] > ``` > Note how a default value of the Python string `bob` is represented in the results of `PRAGMA table_info()` as `default_value="'bob'"` - it's got single quotes added to it! > > So comparing default values from introspecting the database needs me to first parse that syntax. This may require a new table introspection method. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/468#issuecomment-1229279539_ | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/475/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
612658444 | MDU6SXNzdWU2MTI2NTg0NDQ= | 109 | table.create_index(..., ignore=True) | 9599 | closed | 0 | 1 | 2020-05-05T14:44:21Z | 2020-05-05T14:46:53Z | 2020-05-05T14:46:53Z | OWNER | Option to silently do nothing if the index already exists. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/109/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1816851056 | I_kwDOCGYnMM5sSvJw | 568 | table.create(..., replace=True) | 9599 | closed | 0 | 7 | 2023-07-22T18:12:22Z | 2023-07-22T19:25:35Z | 2023-07-22T19:15:44Z | OWNER | Found myself using this pattern to quickly prototype a schema: ```python import sqlite_utils db = sqlite_utils.Database(memory=True) print(db["answers_chunks"].create({ "id": int, "content": str, "embedding_type_id": int, "embedding": bytes, "embedding_content_md5": str, "source": str, }, pk="id", transform=True).schema) ``` <img width="472" alt="image" src="https://github.com/simonw/datasette-answers/assets/9599/30237f7c-4964-4190-984c-d824c57939a9"> Using `replace=True` to drop and then recreate the table would be neat here, and would be consistent with other places that use `replace=True`. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/568/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1485017981 | I_kwDODEpn8M5Yg5N9 | 2 | table identifications has no column named previous_observation_taxon | 520541 | open | 0 | 0 | 2022-12-08T16:47:17Z | 2022-12-08T16:47:17Z | NONE | Installed successfully with pip and ran `inaturalist-to-sqlite inaturalist.db simonw` and got the error: ``` sqlite3.OperationalError: table identifications has no column named previous_observation_taxon ``` | 206202864 | issue | { "url": "https://api.github.com/repos/dogsheep/inaturalist-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
975158266 | MDU6SXNzdWU5NzUxNTgyNjY= | 19 | table activity_summary has no column named appleMoveTime | 9599 | closed | 0 | 0 | 2021-08-20T00:46:44Z | 2021-08-20T00:54:34Z | 2021-08-20T00:54:34Z | MEMBER | Got this error today against a fresh export: table activity_summary has no column named appleMoveTime | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
923602693 | MDU6SXNzdWU5MjM2MDI2OTM= | 276 | support small help flag -h | 601708 | closed | 0 | 0 | 2021-06-17T07:59:31Z | 2021-06-18T14:56:59Z | 2021-06-18T14:56:59Z | CONTRIBUTOR | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/276/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
836829560 | MDU6SXNzdWU4MzY4Mjk1NjA= | 248 | support for Apache Arrow / parquet files I/O | 649467 | open | 0 | 1 | 2021-03-20T14:59:30Z | 2021-10-28T23:46:48Z | NONE | I just started looking at Apache Arrow using pyarrow for import and export of tabular datasets, and it looks quite compelling. It might be worth looking at for sqlite-utils and/or datasette. As a test, I took a random jsonl data dump of a dataset I have with floats, strings, and ints and converted it to arrow's parquet format using the naive `pyarrow.parquet.write_file()` command, which has automatic type inferrence. It compressed down to 7% of the original size. Conversion of a 26MB JSON file and serializing it to parquet was eyeblink instantaneous. Parquet files are portable and can be directly imported into pandas and other analytics software. The only hangup is the automatic type inference of the naive reader. It's great for general laziness and for parsing JSON columns (it correctly interpreted a table of mine with a JSON array). However, I did get an exception for a string column where most entries looked integer-like but had a couple values that weren't -- the reader tried to coerce all of them for some reason, even though the JSON type is string. Since the writer optionally takes a schema, it shouldn't be too hard to grab the sqlite header types. With some additional hinting, you might get datetime columns and JSON, which are native Arrow types. Somewhat tangentially, someone even wrote an sqlite vfs extension for Parquet: https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/248/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
988553806 | MDU6SXNzdWU5ODg1NTM4MDY= | 1457 | suggestion: distinguish names in `--static` documentation | 51016 | closed | 0 | 0 | 2021-09-05T17:04:27Z | 2021-10-14T18:39:55Z | 2021-10-14T18:39:55Z | CONTRIBUTOR | Over in https://docs.datasette.io/en/stable/custom_templates.html#serving-static-files, there is the slightly comical example command - ``` datasette -m metadata.json --static static:static/ --memory ``` (now, with MORE STATIC!) It took me a while to sort out all the URLs and paths involved because I wasn't being very clever. But in the interests of simplification and distinction, I might suggest something like ``` datasette -m metadata.json --static loc:static-files/ --memory ``` I will submit a PR for your consideration. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1457/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
988556488 | MDU6SXNzdWU5ODg1NTY0ODg= | 1459 | suggestion: allow `datasette --open` to take a relative URL | 51016 | open | 0 | 1 | 2021-09-05T17:17:07Z | 2021-09-05T19:59:15Z | CONTRIBUTOR | (soft suggestion because I'm not sure I'm using datasette right yet) Over at https://github.com/ctb/2021-sourmash-datasette, I'm playing around with datasette, and I'm creating some static pages to send people to the right facets. There may well be better ways of achieving this end goal, and I will find out if so, I'm sure! But regardless I think it might be neat to support an option to allow `-o/--open` to take a relative URL, that then gets appended to the hostname and port. This would let me improve my documentation. I don't see any downsides, either, but 🤷 there may well be some :) Happy to dig in and provide a PR if it's of interest. I'm not sure off the top of my head how to support an optional value to a parameter in argparse - the current `-o` behavior is kinda nice so it'd be suboptimal to require a url for `-o`. Maybe `--open-url=` or something would work? | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1459/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
606720674 | MDU6SXNzdWU2MDY3MjA2NzQ= | 736 | strange behavior using accented characters | 30607 | closed | 0 | 3 | 2020-04-25T08:34:51Z | 2020-04-28T06:09:28Z | 2020-04-27T18:59:16Z | NONE | Hi, when I search `incompatibilità` [here](https://my-database.now.sh/commissioniComunePalermo/youtube), using full text search, it becomes `incompatibilitÃÂ ` and I have no result. If I encode the `à` char in the URL (`incompatibilit%C3%A0`) I have the right result. ![image](https://user-images.githubusercontent.com/30607/80275201-00a79380-86e0-11ea-865e-f7e1474e8098.png) | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/736/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
2029161033 | I_kwDOCGYnMM548opJ | 606 | str and int as aliases for text and integer | 9599 | closed | 0 | 2 | 2023-12-06T18:35:49Z | 2023-12-06T19:44:04Z | 2023-12-06T18:49:32Z | OWNER | I keep making this mistake: ```bash sqlite-utils add-column content.db assets _since int ``` ``` Usage: sqlite-utils add-column [OPTIONS] PATH TABLE COL_NAME [[integer|float|b lob|text|INTEGER|FLOAT|BLOB|TEXT]] Try 'sqlite-utils add-column -h' for help. Error: Invalid value for '[[integer|float|blob|text|INTEGER|FLOAT|BLOB|TEXT]]': 'int' is not one of 'integer', 'float', 'blob', 'text', 'INTEGER', 'FLOAT', 'BLOB', 'TEXT'. ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/606/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
503085013 | MDU6SXNzdWU1MDMwODUwMTM= | 13 | statuses-lookup command | 9599 | closed | 0 | 1 | 2019-10-06T11:00:20Z | 2019-10-07T00:33:49Z | 2019-10-07T00:31:44Z | MEMBER | For bulk retrieving tweets by their ID. https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-lookup Rate limit is 900/15 minutes (1 call per second) but each call can pull up to 100 IDs, so we can pull 6,000 per minute. Should support `--SQL` and `--attach` #8 | 206156866 | issue | { "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
707849175 | MDU6SXNzdWU3MDc4NDkxNzU= | 974 | static assets and favicon aren't cached by the browser | 45416 | open | 0 | 1 | 2020-09-24T04:44:55Z | 2022-01-13T22:21:28Z | NONE | Using datasette to solve some frustrating problems with our fulfillment provider today, I was surprised to see repeated requests for assets under /-/static and the favicon. While it won't likely be a huge performance bottleneck, I bet datasette would feel a bit zippier if you had Uvicorn serving up some caching-related headers telling the browser it was safe to cache static assets. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/974/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
637342551 | MDU6SXNzdWU2MzczNDI1NTE= | 834 | startup() plugin hook | 9599 | closed | 0 | 5533512 | 6 | 2020-06-11T21:48:14Z | 2020-06-28T19:38:50Z | 2020-06-13T17:56:12Z | OWNER | It might be useful to have an `startup` hook which gets passed the `datasette` object as soon as Datasette has finished initializing. My initial use-case for this is configuration verification - checking that the `"plugins"` configuration block for this plugin contains valid details. I imagine there are plenty of other potential uses for this as well. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/834/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
516763727 | MDExOlB1bGxSZXF1ZXN0MzM1OTgwMjQ2 | 8 | stargazers command, refs #4 | 9599 | closed | 0 | 5 | 2019-11-03T00:37:36Z | 2020-05-02T20:00:27Z | 2020-05-02T20:00:26Z | MEMBER | dogsheep/github-to-sqlite/pulls/8 | Needs tests. Refs #4. | 207052882 | pull | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
610517472 | MDU6SXNzdWU2MTA1MTc0NzI= | 103 | sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns | 32605365 | closed | 0 | 8 | 2020-05-01T02:26:14Z | 2020-05-14T00:18:57Z | 2020-05-14T00:18:57Z | CONTRIBUTOR | If using insert_all to put in 1000 rows of data with varying number of columns, it comes up with this message `sqlite3.OperationalError: too many SQL variables` if the number of columns is larger in later records (past the first row) I've reduced `SQLITE_MAX_VARS` by 100 to 899 at the top of `db.py` to add wiggle room, so that if the column count increases it wont go past SQLite's batch limit as calculated by this line of code based on the count of the first row's dict keys batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns)) | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
504238461 | MDU6SXNzdWU1MDQyMzg0NjE= | 6 | sqlite3.OperationalError: table users has no column named bio | 1055831 | closed | 0 | 2 | 2019-10-08T19:39:52Z | 2019-10-13T05:31:28Z | 2019-10-13T05:30:19Z | NONE | ``` $ github-to-sqlite repos github.db $ github-to-sqlite starred github.db dazzag24 Traceback (most recent call last): File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/bin/github-to-sqlite", line 10, in <module> sys.exit(cli()) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/cli.py", line 106, in starred utils.save_stars(db, user, stars) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 177, in save_stars user_id = save_user(db, user) File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 61, in save_user return db["users"].upsert(to_save, pk="id").last_pk File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 1067, in upsert extracts=extracts, File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 916, in insert extracts=extracts, File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 1024, in insert_all result = self.db.conn.execute(sql, values)… | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1123393829 | I_kwDODFE5qs5C9aEl | 10 | sqlite3.OperationalError: no such table: main.my_activity | 69208826 | open | 0 | 1 | 2022-02-03T17:59:29Z | 2022-03-20T02:38:07Z | NONE | Hello, When i run the command `google-takeout-to-sqlite my-activity db.db takeout-20220203T174446Z-001.zip`, i get this error : ``` Traceback (most recent call last): File "c:\users\julie\appdata\local\programs\python\python39-32\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "c:\users\julie\appdata\local\programs\python\python39-32\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\julie\AppData\Local\Programs\Python\Python39-32\Scripts\google-takeout-to-sqlite.exe\__main__.py", line 7, in <module> File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py", line 1053, in main rv = self.invoke(ctx) File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\click\core.py", line 754, in invoke return __callback(*args, **kwargs) File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\google_takeout_to_sqlite\cli.py", line 31, in my_activity utils.save_my_activity(db, zf) File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\google_takeout_to_sqlite\utils.py", line 19, in save_my_activity db["my_activity"].create_index(["time"]) File "c:\users\julie\appdata\local\programs\python\python39-32\lib\site-packages\sqlite_utils\db.py", line 629, in create_index self.db.conn.execute(sql) sqlite3.OperationalError: no such table: main.my_activity ``` Thank you for your help … | 206649770 | issue | { "url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
760960559 | MDU6SXNzdWU3NjA5NjA1NTk= | 205 | sqlite3.OperationalError: near "(": syntax error | 765871 | closed | 0 | 2 | 2020-12-10T06:44:40Z | 2020-12-10T19:18:22Z | 2020-12-10T07:24:23Z | NONE | The sqlite version is 3.22.0 2018-01-22 18:45:57 0c55d179733b46d8d0ba4d88e01a25e10677046ee3da1d5b1581e86726f2alt1 sqlite-utils, version 3.0 It fails here: https://github.com/kaihendry/aws-partners-datasette/runs/1528432635?check_suite_focus=true I'm not sure where the problem is, since it works _fine locally_ on Archlinux system running 3.34.0 2020-12-01 16:14:00 a26b6597e3ae272231b96f9982c3bcc17ddec2f2b6eb4df06a224b91089fed5b https://github.com/kaihendry/aws-partners-datasette/blob/main/create-summary-view.sh Maybe I need to bump up from ubuntu-latest to ? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/205/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
709577625 | MDU6SXNzdWU3MDk1Nzc2MjU= | 179 | sqlite-utils transform/insert --detect-types | 9599 | closed | 0 | 4 | 2020-09-26T17:28:55Z | 2021-06-19T03:36:16Z | 2021-06-19T03:36:05Z | OWNER | Idea from https://github.com/simonw/datasette-edit-tables/issues/13 - provide Python utility methods and accompanying CLI options for detecting the likely types of TEXT columns. So if you have a text column that actually contained exclusively integer string values, it can let you know and let you run transform against it. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/179/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
706017416 | MDU6SXNzdWU3MDYwMTc0MTY= | 164 | sqlite-utils transform sub-command | 9599 | closed | 0 | 5897911 | 4 | 2020-09-22T01:32:20Z | 2020-09-24T20:34:50Z | 2020-09-22T07:48:05Z | OWNER | The `.transform()` method in #114 warrants an equivalent CLI tool. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/164/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
708293114 | MDU6SXNzdWU3MDgyOTMxMTQ= | 176 | sqlite-utils transform column order option | 9599 | closed | 0 | 2 | 2020-09-24T16:01:21Z | 2020-09-24T20:34:51Z | 2020-09-24T16:11:59Z | OWNER | Split from #175 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/176/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
761915790 | MDU6SXNzdWU3NjE5MTU3OTA= | 206 | sqlite-utils should suggest --csv if JSON parsing fails | 9599 | closed | 0 | 4 | 2020-12-11T05:17:56Z | 2021-10-30T15:52:17Z | 2021-01-03T18:42:22Z | OWNER | ``` ~ % gsutil cat gs://ossf-criticality-score/python_top_200.csv | sqlite-utils insert /tmp/crit.db crit - ... File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) ``` A nicer error message here would be one that says the JSON is invalid but suggests that maybe you could try `--csv`. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/206/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
735532751 | MDU6SXNzdWU3MzU1MzI3NTE= | 192 | sqlite-utils search command | 9599 | closed | 0 | 6079500 | 9 | 2020-11-03T18:07:59Z | 2020-11-08T17:07:01Z | 2020-11-08T17:07:01Z | OWNER | A command that knows how to run a search against a FTS enabled table and return results ranked by relevance. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/192/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
738115165 | MDU6SXNzdWU3MzgxMTUxNjU= | 200 | sqlite-utils rows -c option | 9599 | closed | 0 | 6079500 | 1 | 2020-11-07T00:22:12Z | 2020-11-07T00:28:48Z | 2020-11-07T00:28:47Z | OWNER | To let you specify the exact columns you want. Based on the `-c` option to `sqlite-utils search` in #192. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/200/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1352931464 | I_kwDOCGYnMM5QpBiI | 469 | sqlite-utils rows --order option | 9599 | closed | 0 | 8355157 | 1 | 2022-08-27T03:49:51Z | 2022-08-27T04:30:49Z | 2022-08-27T04:10:32Z | OWNER | For consistency with `search`: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#search ``` -o, --order TEXT Order by ('column' or 'column desc') ``` I wanted to run `sqlite-utils rows db.db mytable --order 'rowid desc'` to see the most recently imported rows. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/469/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
665802405 | MDU6SXNzdWU2NjU4MDI0MDU= | 124 | sqlite-utils query should support named parameters | 9599 | closed | 0 | 1 | 2020-07-26T15:25:10Z | 2020-07-30T22:57:51Z | 2020-07-27T03:53:58Z | OWNER | To help out with escaping - so you can run this: sqlite-utils query "insert into foo (blah) values (:blah)" --param blah `something here` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/124/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1352932716 | I_kwDOCGYnMM5QpB1s | 471 | sqlite-utils query --functions mechanism for registering extra functions | 9599 | closed | 0 | 8355157 | 12 | 2022-08-27T03:57:53Z | 2022-09-07T03:46:26Z | 2022-08-27T05:10:57Z | OWNER | It would be really cool if you could register additional custom SQL functions for use with the `sqlite-utils query` command - something like this: ``` sqlite-utils data.db 'update images set domain = extract_domain(url)' --functions ' from urllib.parse import urlparse def extract_domain(url): return urlparse(url).netloc ' ``` Every function defined in that code block would be registered with the connection, unless the name began with an underscore. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
924990677 | MDU6SXNzdWU5MjQ5OTA2Nzc= | 279 | sqlite-utils memory should handle TSV and JSON in addition to CSV | 9599 | closed | 0 | 7 | 2021-06-18T15:02:54Z | 2021-06-19T03:11:59Z | 2021-06-19T03:11:59Z | OWNER | - Use sniff to detect CSV or TSV (if `:tsv` or `:csv` was not specified) and delimiters Follow-on from #272 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
922099793 | MDExOlB1bGxSZXF1ZXN0NjcxMDE0NzUx | 273 | sqlite-utils memory command for directly querying CSV/JSON data | 9599 | closed | 0 | 8 | 2021-06-16T05:04:58Z | 2021-06-18T15:01:17Z | 2021-06-18T15:00:52Z | OWNER | simonw/sqlite-utils/pulls/273 | Refs #272. Initial implementation only does CSV data, still needs: - [x] Implement `--save` - [x] Add `--dump` to the documentation - [x] Add `--attach` example to the documentation - [x] Replace `:memory:` in documentation | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
990844088 | MDU6SXNzdWU5OTA4NDQwODg= | 325 | sqlite-utils memory can't deal with multiple files with the same name | 144773 | closed | 0 | 4 | 2021-09-08T08:14:42Z | 2021-09-22T20:52:56Z | 2021-09-22T20:45:45Z | NONE | When I use multiple files with the same name, e.g. in `sqlite-utils memory a/bug.csv b/bug.csv`, sqlite-utils creates invalid views. ``` Traceback (most recent call last): File "/home/karl/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 763, in invoke return __callback(*args, **kwargs) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/cli.py", line 1299, in memory db[csv_table].transform(types=tracker.types) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/db.py", line 1287, in transform self.db.execute(sql) File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/db.py", line 421, in execute return self.conn.execute(sql) sqlite3.OperationalError: error in view t1: no such table: main.bug ``` This can be reproduced with ```sh #!/bin/bash mkdir foo mkdir bar echo -e 'col1,col2\nval1,val2' > foo/bug.csv echo -e 'col3,col4\nval3,val4' > bar/bug.csv sqlite-utils memory */bug.csv 'SELECT 1' ``` Ideally, the tables would get unique names by including the next path segment until the names are unique. But just making the numbered t* aliases work would be good enough. This problem can of course be worked around by… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/325/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
925545468 | MDU6SXNzdWU5MjU1NDU0Njg= | 288 | sqlite-utils memory blah.json --schema | 9599 | closed | 0 | 0 | 2021-06-20T08:10:40Z | 2021-06-20T18:26:21Z | 2021-06-20T18:26:21Z | OWNER | Like `--dump` but only outputs the schema - useful for understanding what you are about to run queries against. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/288/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
976405225 | MDU6SXNzdWU5NzY0MDUyMjU= | 320 | sqlite-utils memory --analyze option | 9599 | closed | 0 | 2 | 2021-08-22T15:37:10Z | 2021-08-22T15:46:56Z | 2021-08-22T15:44:29Z | OWNER | To provide a way of running [analyze-tables](https://sqlite-utils.datasette.io/en/stable/cli.html#analyzing-tables) directly against JSON or CSV data. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/320/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
675753042 | MDU6SXNzdWU2NzU3NTMwNDI= | 131 | sqlite-utils insert: options for column types | 9599 | open | 0 | 5 | 2020-08-09T18:59:11Z | 2022-03-15T13:21:42Z | OWNER | The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs. It would be useful if you could do the following: - automatically detects the column types based on eg the first 1000 records - explicitly state the rule for specific columns `--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types` For specific columns maybe this: sqlite-utils insert db.db images images.tsv \ --tsv \ -c id int \ -c score float | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1920416843 | I_kwDOCGYnMM5ydzxL | 597 | sqlite-utils insert-files should be able to convert fields | 1737541 | open | 0 | 0 | 2023-09-30T22:20:47Z | 2023-09-30T22:20:47Z | NONE | Currently using both `insert-files` and `convert` is needed in order to create sqlar files, it would be more convenient if it could be done with just one command. ```shell ~ ❯ cat test.py import os class Example: def __init__(self, arg1, arg2): self.arg1 = arg1 ~ ❯ sqlite-utils insert-files test.sqlar sqlar test.py -c name:name -c data:content -c mode:mode -c mtime:mtime -c sz:size --pk=name [####################################] 100% ~ ❯ sqlite-utils convert test.sqlar sqlar data "zlib.compress(value)" --import=zlib --where "name = 'test.py'" [####################################] 100% ~ ❯ cat test.py | sqlite-utils convert test.sqlar sqlar data "zlib.compress(sys.stdin.buffer.read())" --import=zlib --import=sys --where "name = 'test.py'" # Alternative way [####################################] 100% ~ ❯ sqlite3 test.sqlar "SELECT hex(data) FROM sqlar WHERE name = 'test.py';" | python3 -c "import sys, zlib; sys.stdout.buffer.write(zlib.decompress(bytes.fromhex(sys.stdin.read())))" import os class Example: def __init__(self, arg1, arg2): self.arg1 = arg1 ~ ❯ rm test.py ~ ❯ sqlar -l test.sqlar test.py ~ ❯ sqlar -x test.sqlar ~ ❯ cat test.py import os class Example: def __init__(self, arg1, arg2): self.arg1 = arg1 ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/597/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
963897111 | MDU6SXNzdWU5NjM4OTcxMTE= | 309 | sqlite-utils insert errors should show SQL and parameters, if possible | 16622642 | closed | 0 | 6 | 2021-08-09T11:24:14Z | 2021-08-09T23:40:29Z | 2021-08-09T22:25:58Z | NONE | I've tried several approaches, but this is the current one: ```sh echo $json-line | sqlite-utils insert json.db jsontable --truncate --alter --detect-types - ``` In all cases, I get this error: ```sh OverflowError: Python int too large to convert to SQLite INTEGER Traceback (most recent call last): File "/home/sean/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 841, in insert insert_upsert_implementation( File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 780, in insert_upsert_implementation db[table].insert_all( File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py", line 2145, in insert_all self.insert_chunk( File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py", line 1957, in insert_chunk result = self.db.execute(query, params) File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py", line 257, in execute return self.conn.execute(sql, parameters) ``` I googled the error and checked SO answers and advice, all good. I changed my JSON file to not use integers so I no longer get this error. Of course, that makes using the database a bit harder, so I also tried to solve the problem by modifying DB structure (while using integers in JSON). If change all `INTEGER` Data Types to something else (`ST… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/309/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
738514367 | MDU6SXNzdWU3Mzg1MTQzNjc= | 202 | sqlite-utils insert -f colname - for configuring full-text search | 9599 | closed | 0 | 2 | 2020-11-08T17:30:09Z | 2021-01-03T05:00:36Z | 2021-01-03T05:00:27Z | OWNER | A mechanism for specifying columns that should be configured for full-text search as part of the initial data import: sqlite-utils insert mydb.db articles articles.csv --csv -f title -f body | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/202/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
470131537 | MDU6SXNzdWU0NzAxMzE1Mzc= | 41 | sqlite-utils insert --tsv option | 9599 | closed | 0 | 0 | 2019-07-19T04:27:21Z | 2019-07-19T04:50:47Z | 2019-07-19T04:50:47Z | OWNER | Right now we only support ingesting CSV, but sometimes interesting data is released as TSV. https://www.washingtonpost.com/national/2019/07/18/how-download-use-dea-pain-pills-database/ for example. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/41/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1042569687 | I_kwDOCGYnMM4-JFnX | 335 | sqlite-utils index-foreign-keys fails due to pre-existing index | 596279 | closed | 0 | 11 | 2021-11-02T16:22:11Z | 2021-11-14T22:55:56Z | 2021-11-14T22:55:56Z | NONE | While running the command: ```sh sqlite-utils index-foreign-keys $SQLITE_DIR/pudl.sqlite ``` I got the following error: ``` Traceback (most recent call last): File "/home/zane/miniconda3/envs/pudl-dev/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/cli.py", line 454, in index_foreign_keys db.index_foreign_keys() File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py", line 902, in index_foreign_keys table.create_index([fk.column]) File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py", line 1563, in create_index self.db.execute(sql) File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py", line 421, in execute return self.conn.execute(sql) sqlite3.OperationalError: index idx_generators_eia860_report_date already exists ``` This DB was created with the foreign key constraint `PRAGMA` enabled and a bunch of column-level `CHECK` constraints. Is this an expected behavior? Should one not try to index foreign keys if FK constraints are already being enforced within the DB? I'm also noticing that the size of the DB after FK … | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/335/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
462423972 | MDExOlB1bGxSZXF1ZXN0MjkzMTE3MTgz | 34 | sqlite-utils index-foreign-keys / db.index_foreign_keys() | 9599 | closed | 0 | 0 | 2019-06-30T16:43:40Z | 2019-06-30T23:50:55Z | 2019-06-30T23:50:55Z | OWNER | simonw/sqlite-utils/pulls/34 | Refs #33 - [x] `sqlite-utils index-foreign-keys` command - [x] `db.index_foreign_keys()` method - [x] unit tests - [x] documentation | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/34/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
816526538 | MDU6SXNzdWU4MTY1MjY1Mzg= | 239 | sqlite-utils extract could handle nested objects | 9599 | open | 0 | 16 | 2021-02-25T15:10:28Z | 2022-09-03T23:46:02Z | OWNER | Imagine a table (imported from a nested JSON file) where one of the columns contains values that look like this: {"email": "anonymous@noreply.airtable.com", "id": "usrROSHARE0000000", "name": "Anonymous"} The `sqlite-utils extract` command already uses single text values in a column to populate a new table. It would not be much of a stretch for it to be able to use JSON instead, including specifying which of those values should be used as the primary key in the new table. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239/reactions", "total_count": 6, "+1": 5, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
769397742 | MDU6SXNzdWU3NjkzOTc3NDI= | 3 | sqlite-utils error on takeout import | 231498 | open | 0 | 0 | 2020-12-17T01:18:48Z | 2020-12-17T01:19:04Z | NONE | ``` $ google-takeout-to-sqlite my-activity takeout.db /path/to/zip ... sqlite3.OperationalError: no such table: main.my_activity ``` there is no table create in `utils.py`, unlike other importers such as github-to-sqlite additionally, this package and hackernews-to-sqlite have conflicting `sqlite-utils` dep with datasette and dogsheep-beta | 206649770 | issue | { "url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/3/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
777530107 | MDU6SXNzdWU3Nzc1MzAxMDc= | 214 | sqlite-utils enable-counts command | 9599 | closed | 0 | 0 | 2021-01-02T21:45:48Z | 2021-01-03T04:26:44Z | 2021-01-03T04:26:44Z | OWNER | The CLI version of #212 and #213. # Enable counts for all tables: sqlite-utils enable-counts data.db # Enable counts for specific tables: sqlite-utils enable-counts data.db table1 table2 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/214/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
922832113 | MDU6SXNzdWU5MjI4MzIxMTM= | 274 | sqlite-utils dump my.db command | 9599 | closed | 0 | 0 | 2021-06-16T16:30:14Z | 2021-06-16T23:51:54Z | 2021-06-16T23:51:54Z | OWNER | Inspired by the `--dump` mechanism I added to `sqlite-utils memory` here: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937 > Can use `.iterdump()` to implement this: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump > > Maybe instead (or as-well-as) offer `--dump` which dumps out the SQL from that. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/274/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
615477131 | MDU6SXNzdWU2MTU0NzcxMzE= | 111 | sqlite-utils drop-table and drop-view commands | 9599 | closed | 0 | 2 | 2020-05-10T21:10:42Z | 2020-05-11T01:58:36Z | 2020-05-11T00:44:26Z | OWNER | Would be useful to be able to drop views and tables from the CLI. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/111/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
611222968 | MDU6SXNzdWU2MTEyMjI5Njg= | 107 | sqlite-utils create-view CLI command | 9599 | closed | 0 | 2 | 2020-05-02T16:15:13Z | 2020-05-03T15:36:58Z | 2020-05-03T15:36:37Z | OWNER | Can go with #27 - `sqlite-utils create-table`. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/107/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
455496504 | MDU6SXNzdWU0NTU0OTY1MDQ= | 27 | sqlite-utils create-table command | 9599 | closed | 0 | 8 | 2019-06-13T01:43:30Z | 2020-05-03T15:26:15Z | 2020-05-03T15:26:15Z | OWNER | Spun off from #24 - it would be useful if CLI users could create new tables (with explicit column types, not null rules and defaults) without having to insert an example record. - [x] Get it working - [x] Support `--pk` - [x] Support `--not-null` - [x] Support `--default` - [x] Support `--fk colname othertable othercol` - [x] Support `--replace` and `--ignore` - [x] Documentation | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1212701569 | I_kwDOCGYnMM5ISFuB | 427 | sqlite-utils convert date parsing recipe complains about trying to parse "*" | 1385831 | closed | 0 | 1 | 2022-04-22T19:27:10Z | 2022-07-02T13:59:59Z | 2022-07-02T13:59:32Z | NONE | Missing values in my dataset are denoted by a single asterisk. I am trying to parse string dates into dates. This works fine for columns without missing values, but, when the column contains "*", I get the following: ``` $ sqlite-utils convert ${dbfile} details dob 'r.parsedate(value)' [------------------------------------] 0%Traceback (most recent call last): File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 2508, in convert_value return fn(v) File "<string>", line 2, in fn File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/recipes.py", line 8, in parsedate parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 1368, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 643, in parse raise ParserError("Unknown string format: %s", timestr) dateutil.parser._parser.ParserError: Unknown string format: * Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 33, in <module> sys.exit(load_entry_point('sqlite-utils==3.25.1', 'console_scripts', 'sqlite-utils')()) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, … | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/427/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
957536983 | MDExOlB1bGxSZXF1ZXN0NzAwOTQ0NjQ0 | 303 | sqlite-utils convert command and db[table].convert(...) method | 9599 | closed | 0 | 1 | 2021-08-01T16:52:42Z | 2021-08-02T04:47:42Z | 2021-08-02T04:47:39Z | OWNER | simonw/sqlite-utils/pulls/303 | Refs #251, #302. - [x] Get recipes working - [x] Document recipes - [x] Implement `db[table].convert(...)` method - [x] Add tests for recipes that use the new Python method - [x] Implement `db[table].convert(..., multi=True)` mechanism - [x] Documentation for `db[table].convert(...)` - [x] Refactor `sqlite-utils convert` to use the new method | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/303/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
763320133 | MDExOlB1bGxSZXF1ZXN0NTM3NzkxNjc1 | 208 | sqlite-utils analyze-tables command and table.analyze_column() method | 9599 | closed | 0 | 6 | 2020-12-12T05:27:49Z | 2020-12-13T07:20:16Z | 2020-12-13T07:20:12Z | OWNER | simonw/sqlite-utils/pulls/208 | Refs #207 - [x] Improve design of CLI output - [x] Truncate long values in least/most common - [x] Add a `-c` column selection option - [x] Tests - [x] Documentation | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/208/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 |