{"html_url": "https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866184260", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/267", "id": 866184260, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NjE4NDI2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T17:26:18Z", "updated_at": "2021-06-22T17:27:27Z", "author_association": "OWNER", "body": "If an`.update()` method doesn't work because it collides with an existing dictionary method a `.pk` property could still be nice:\r\n```python\r\nfor row in db[\"sometable\"].rows:\r\n db[\"sometable\"].update(row.pk, {\"modified\": 1})\r\n```\r\n \r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 915421499, "label": "row.update() or row.pk"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866182655", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/267", "id": 866182655, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NjE4MjY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T17:24:03Z", "updated_at": "2021-06-22T17:24:03Z", "author_association": "OWNER", "body": "I'm re-opening this as a research task because it may be possible to cleanly implement this using a `dict` subclass - some notes on that here: https://treyhunner.com/2019/04/why-you-shouldnt-inherit-from-list-and-dict-in-python/\r\n\r\nSince this would just be for adding methods (and maybe a property for returning the primary keys for a row) the usual disadvantages of subclassing `dict` described in that article shouldn't apply.\r\n\r\nOne catch: dictionaries already have a `.update()` method! So would have to pick another name.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 915421499, "label": "row.update() or row.pk"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865510796", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290", "id": 865510796, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTUxMDc5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T04:04:40Z", "updated_at": "2021-06-22T04:04:48Z", "author_association": "OWNER", "body": "Still needs documentation, which will involve rewriting the whole [Executing queries](https://sqlite-utils.datasette.io/en/3.11/python-api.html#executing-queries) section.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 926777310, "label": "`db.query()` method (renamed `db.execute_returning_dicts()`)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865497846", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290", "id": 865497846, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTQ5Nzg0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T03:21:38Z", "updated_at": "2021-06-22T03:21:38Z", "author_association": "OWNER", "body": "The Python docs say: https://docs.python.org/3/library/sqlite3.html\r\n\r\n> To retrieve data after executing a SELECT statement, you can either treat the cursor as an iterator, call the cursor\u2019s `fetchone()` method to retrieve a single matching row, or call `fetchall()` to get a list of the matching rows.\r\n\r\nLooking at the C source code, both `fetchmany()` and `fetchall()` work under the hood by assembling a Python list: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L907-L972 - see calls to `PyList_Append()`\r\n\r\nSo it looks like the most efficient way to iterate over a cursor may well be `for row in cursor:` - which I think calls this C function: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L813-L876", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 926777310, "label": "`db.query()` method (renamed `db.execute_returning_dicts()`)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865495370", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290", "id": 865495370, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTQ5NTM3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T03:14:30Z", "updated_at": "2021-06-22T03:14:30Z", "author_association": "OWNER", "body": "One small problem with the existing method:\r\nhttps://github.com/simonw/sqlite-utils/blob/8cedc6a8b29180e68326f6b76f249d5e39e4b591/sqlite_utils/db.py#L362-L365\r\n\r\nIt returns a full list, but what if the user would rather have a generator they can iterate over without loading the results into memory in one go?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 926777310, "label": "`db.query()` method (renamed `db.execute_returning_dicts()`)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865491922", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290", "id": 865491922, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTQ5MTkyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T03:05:35Z", "updated_at": "2021-06-22T03:05:35Z", "author_association": "OWNER", "body": "Potential names:\r\n\r\n- `db.query(sql)` - it's weird to have both this and `db.execute()` but it is at least short and memorable\r\n- `db.sql(sql)`\r\n- `db.execute_d(sql)` - ugly\r\n- `db.execute_dicts(sql)` - confusing\r\n- `db.execute_sql(sql)` - easily confused with `db.execute(sql)`\r\n\r\nI think `db.query(sql)` may be the best option here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 926777310, "label": "`db.query()` method (renamed `db.execute_returning_dicts()`)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1368#issuecomment-865204472", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1368", "id": 865204472, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTIwNDQ3Mg==", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-06-21T17:11:37Z", "updated_at": "2021-06-21T17:11:37Z", "author_association": "CONTRIBUTOR", "body": "If this is a concept ACK then I will move onto fixing the tests (adding new ones) and updating the documentation for the new plugin hook.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 913865304, "label": "DRAFT: A new plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1368#issuecomment-865160132", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1368", "id": 865160132, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTE2MDEzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-21T16:07:06Z", "updated_at": "2021-06-21T16:08:48Z", "author_association": "OWNER", "body": "A few tests failed - Black, the test that checks the docs mention the new hook - the most interesting failing test looks like this one:\r\n```\r\n updated_metadata[\"databases\"][\"fixtures\"][\"queries\"][\"magic_parameters\"][\r\n \"allow\"\r\n ] = (allow if \"query\" in permissions else deny)\r\n> cascade_app_client.ds._metadata = updated_metadata\r\nE AttributeError: can't set attribute\r\n```\r\nFrom https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_permissions.py#L439-L467\r\n\r\nThis test is directly manipulating `_metadata` purely for the purposes of simulating different permissions - I think updating it to manipulate `_local_metadata` instead would fix that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 913865304, "label": "DRAFT: A new plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864621099", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/278", "id": 864621099, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDYyMTA5OQ==", "user": {"value": 601708, "label": "mcint"}, "created_at": "2021-06-20T22:39:57Z", "updated_at": "2021-06-20T22:39:57Z", "author_association": "CONTRIBUTOR", "body": "Fair. I looked into it, it looks like it could be done, but it would be _a bit ugly_. I can upload and link a gist of my exploration. **Click** can parse a first argument while still recognizing it as a sub-command keyword. From there, the program could:\r\n1. ignore it preemptively if it matches a sub-command\r\n2. and/or check if a (db) file exists at the path.\r\n\r\nIt would then also need to set a shared db argument variable.\r\n\r\nClick also makes it easy to parse arguments from environment variables. If you're amenable, I may submit a patch for only that, which would update each sub-command to check for a DB/SQLITE_UTILS_DB environment variable. The goal would be usage that looks like: `DB=./convenient.db sqlite-utils [operation] [args]`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923697888, "label": "Support db as first parameter before subcommand, or as environment variable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/289#issuecomment-864609271", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/289", "id": 864609271, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDYwOTI3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-20T20:42:07Z", "updated_at": "2021-06-20T20:42:07Z", "author_association": "OWNER", "body": "Wow, thank you! I didn't know about `typing.cast()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925677191, "label": "Mypy fixes for rows_from_file()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/286#issuecomment-864594956", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/286", "id": 864594956, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDU5NDk1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-20T18:38:05Z", "updated_at": "2021-06-20T18:38:05Z", "author_association": "OWNER", "body": "3.10 is out in Homebrew now (they turn that around so fast): https://formulae.brew.sh/formula/sqlite-utils", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925487946, "label": "Add installation instructions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1382#issuecomment-864480051", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1382", "id": 864480051, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQ4MDA1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-20T00:20:06Z", "updated_at": "2021-06-20T00:21:02Z", "author_association": "OWNER", "body": "Yes you can - thanks for pointing this out, I've added a comment to the `install.sh` script in the `datasette-csvs` Glitch project:\r\n\r\n```bash\r\npip3 install -U --no-cache-dir -r requirements.txt --user && \\\r\n mkdir -p .data && \\\r\n rm .data/data.db || true && \\\r\n for f in *.csv\r\n do\r\n # Add --encoding=latin-1 to the following if your CSVs use a different encoding:\r\n sqlite-utils insert .data/data.db ${f%.*} $f --csv\r\n done\r\n```\r\n\r\nSo if you edit that file in your own project and change the line to this:\r\n\r\n sqlite-utils insert .data/data.db ${f%.*} $f --csv --encoding=iso-8859-1\r\n\r\nIt should fix this for you.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925406964, "label": "Datasette with Glitch - is it possible to use CSV with ISO-8859-1 encoding?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864476167", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 864476167, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQ3NjE2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T23:36:48Z", "updated_at": "2021-06-19T23:36:48Z", "author_association": "OWNER", "body": "Wrote this up on my blog here: https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/ - with a video demo here: https://www.youtube.com/watch?v=OUjd0rkc678", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864419283", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/284", "id": 864419283, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxOTI4Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:15:34Z", "updated_at": "2021-06-19T15:15:34Z", "author_association": "OWNER", "body": "I think this code is at fault: https://github.com/simonw/sqlite-utils/blob/5b257949d996fe43dc5d218d4308b88796a90740/sqlite_utils/db.py#L1017-L1023\r\n\r\nIt's using `.pks` which adds `rowid` if it's missing.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925320167, "label": ".transform(types=) turns rowid into a concrete column"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418795", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864418795, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxODc5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:11:05Z", "updated_at": "2021-06-19T15:11:14Z", "author_association": "OWNER", "body": "Actually I'm going to go with `use_rowid` instead - because the table doesn't inherently use a rowid itself, but you should use one if you want to query it in a way that gives you back a primary key.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418188", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864418188, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxODE4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:05:53Z", "updated_at": "2021-06-19T15:05:53Z", "author_association": "OWNER", "body": "```python\r\n @property\r\n def uses_rowid(self):\r\n return not any(column for column in self.columns if column.is_pk)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417808", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864417808, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNzgwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:03:00Z", "updated_at": "2021-06-19T15:03:00Z", "author_association": "OWNER", "body": "I think I like `table.uses_rowid` best - it reads well.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417765", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864417765, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNzc2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:02:42Z", "updated_at": "2021-06-19T15:02:42Z", "author_association": "OWNER", "body": "Some options:\r\n\r\n- `table.rowid_only`\r\n- `table.rowid_as_pk`\r\n- `table.no_pks`\r\n- `table.no_pk`\r\n- `table.uses_rowid`\r\n- `table.use_rowid`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417493", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864417493, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNzQ5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:00:43Z", "updated_at": "2021-06-19T15:00:43Z", "author_association": "OWNER", "body": "I have to be careful about the language I use here. Here's the official definition: https://www.sqlite.org/rowidtable.html\r\n\r\n> A \"rowid table\" is any table in an SQLite schema that\r\n> \r\n> - is *not* a [virtual table](https://www.sqlite.org/vtab.html), and\r\n> - is *not* a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table.\r\n> \r\n> Most tables in a typical SQLite database schema are rowid tables.\r\n> \r\n> Rowid tables are distinguished by the fact that they all have a unique, non-NULL, signed 64-bit integer [rowid](https://www.sqlite.org/lang_createtable.html#rowid) that is used as the access key for the data in the underlying [B-tree](https://www.sqlite.org/fileformat2.html#btree) storage engine.\r\n\r\nSo it's not correct to call a table a \"rowid table\" only if it is missing its own primary keys.\r\n\r\nMaybe `table.has_rowid` is the right language to use here? No, that's no good - because tables with their own primary keys usually also have a rowid.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417133", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864417133, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNzEzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T14:57:36Z", "updated_at": "2021-06-19T14:57:36Z", "author_association": "OWNER", "body": "So the logic is:\r\n```python\r\n[column.name for column in self.columns if column.is_pk]\r\n```\r\nI need to decide on a property name. Existing names are documented here: https://sqlite-utils.datasette.io/en/stable/python-api.html#introspection", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417031", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864417031, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNzAzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T14:56:45Z", "updated_at": "2021-06-19T14:56:45Z", "author_association": "OWNER", "body": "```pycon\r\n>>> db = sqlite_utils.Database(memory=True)\r\n>>> db[\"rowid_table\"].insert({\"name\": \"Cleo\"})\r\n