github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/issues/1179#issuecomment-755492945 | https://api.github.com/repos/simonw/datasette/issues/1179 | 755492945 | MDEyOklzc3VlQ29tbWVudDc1NTQ5Mjk0NQ== | 9599 | 2021-01-06T18:37:39Z | 2021-01-06T18:37:39Z | OWNER | I think I'll call this `full_path` for consistency with Django. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780278550 | |
https://github.com/simonw/datasette/issues/1179#issuecomment-755489974 | https://api.github.com/repos/simonw/datasette/issues/1179 | 755489974 | MDEyOklzc3VlQ29tbWVudDc1NTQ4OTk3NA== | 9599 | 2021-01-06T18:35:24Z | 2021-01-06T18:35:24Z | OWNER | Django calls this ` HttpRequest.get_full_path()` - for the path plus the querystring. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780278550 | |
https://github.com/simonw/datasette/issues/1179#issuecomment-755486103 | https://api.github.com/repos/simonw/datasette/issues/1179 | 755486103 | MDEyOklzc3VlQ29tbWVudDc1NTQ4NjEwMw== | 9599 | 2021-01-06T18:32:41Z | 2021-01-06T18:34:11Z | OWNER | This parameter will return the URL path, with querystring arguments, to the HTML version of the page - e.g. `/github/issue_comments` or `/github/issue_comments?_sort_desc=created_at` Open questions: - What should it be called? `path` could be misleading since it also includes the querystring. - Should I provide a `url` or `full_url` version which includes `https://blah.com/...`? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780278550 | |
https://github.com/simonw/datasette/issues/782#issuecomment-755484384 | https://api.github.com/repos/simonw/datasette/issues/782 | 755484384 | MDEyOklzc3VlQ29tbWVudDc1NTQ4NDM4NA== | 9599 | 2021-01-06T18:31:14Z | 2021-01-06T18:31:57Z | OWNER | In building https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on I discovered the following patterns for importing data into both Pandas and Observable/d3: ```python import pandas df = pandas.read_json( "https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array" ) ``` And: ```javascript d3 = require("d3@5") rows = d3.json( "https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array" ) ``` Once again I find myself torn on the best possible default. A list of JSON objects is instantly compatible with both `pandas.read_json()` and `d3.json()` - but it leaves nowhere to put the extra information like pagination and suchlike! Even given this I still think the correct default is an object with `"rows"`, `"total"` and `"next_url"` keys. I should commit to that and implement it - this thought exercise has been running for far too long. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
627794879 | |
https://github.com/simonw/datasette/issues/1178#issuecomment-755476820 | https://api.github.com/repos/simonw/datasette/issues/1178 | 755476820 | MDEyOklzc3VlQ29tbWVudDc1NTQ3NjgyMA== | 9599 | 2021-01-06T18:24:47Z | 2021-01-06T18:24:47Z | OWNER | Issue fixed - https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on displays the correct schemes now. I can't think of a reason anyone on Cloud Run would ever NOT want the `force_https_urls` option, but just in case I've made it so if you pass `--extra-options --setting force_https_urls off` to `publish cloudrun` your setting will be respected. https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/publish/cloudrun.py#L105-L110 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780267857 | |
https://github.com/simonw/datasette/issues/1178#issuecomment-755468795 | https://api.github.com/repos/simonw/datasette/issues/1178 | 755468795 | MDEyOklzc3VlQ29tbWVudDc1NTQ2ODc5NQ== | 9599 | 2021-01-06T18:14:35Z | 2021-01-06T18:14:35Z | OWNER | Deploying that change now to test it. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780267857 | |
https://github.com/simonw/datasette/issues/1178#issuecomment-755163886 | https://api.github.com/repos/simonw/datasette/issues/1178 | 755163886 | MDEyOklzc3VlQ29tbWVudDc1NTE2Mzg4Ng== | 9599 | 2021-01-06T08:37:51Z | 2021-01-06T08:37:51Z | OWNER | Easiest fix would be for `publish cloudrun` to set `force_https_urls`: `datasette publish now` used to do this: https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/publish/now.py#L59-L63 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780267857 | |
https://github.com/simonw/datasette/issues/1179#issuecomment-755161574 | https://api.github.com/repos/simonw/datasette/issues/1179 | 755161574 | MDEyOklzc3VlQ29tbWVudDc1NTE2MTU3NA== | 9599 | 2021-01-06T08:32:31Z | 2021-01-06T08:32:31Z | OWNER | An optional `path` argument to https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette which shows the path WITHOUT the `.Notebook` extension would be useful here. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780278550 | |
https://github.com/simonw/datasette/issues/1178#issuecomment-755160187 | https://api.github.com/repos/simonw/datasette/issues/1178 | 755160187 | MDEyOklzc3VlQ29tbWVudDc1NTE2MDE4Nw== | 9599 | 2021-01-06T08:29:35Z | 2021-01-06T08:29:35Z | OWNER | https://latest-with-plugins.datasette.io/-/asgi-scope ``` {'asgi': {'spec_version': '2.1', 'version': '3.0'}, 'client': ('169.254.8.129', 54971), 'headers': [(b'host', b'latest-with-plugins.datasette.io'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko' b'/20100101 Firefox/84.0'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'webp,*/*;q=0.8'), (b'accept-language', b'en-US,en;q=0.5'), (b'dnt', b'1'), (b'cookie', b'_ga_LL6M7BK6D4=GS1.1.1609886546.49.1.1609886923.0; _ga=GA1.1' b'.894633707.1607575712'), (b'upgrade-insecure-requests', b'1'), (b'x-client-data', b'CgSL6ZsV'), (b'x-cloud-trace-context', b'e776af843c657d2a3da28a73b726e6fe/14187666787557102189;o=1'), (b'x-forwarded-for', b'148.64.98.14'), (b'x-forwarded-proto', b'https'), (b'forwarded', b'for="148.64.98.14";proto=https'), (b'accept-encoding', b'gzip, deflate, br'), (b'content-length', b'0')], 'http_version': '1.1', 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', 'root_path': '', 'scheme': 'http', 'server': ('169.254.8.130', 8080), 'type': 'http'} ``` Note the `'scheme': 'http'` but also the `(b'x-forwarded-proto', b'https')`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780267857 | |
https://github.com/simonw/datasette/issues/1176#issuecomment-755159583 | https://api.github.com/repos/simonw/datasette/issues/1176 | 755159583 | MDEyOklzc3VlQ29tbWVudDc1NTE1OTU4Mw== | 9599 | 2021-01-06T08:28:20Z | 2021-01-06T08:28:20Z | OWNER | I used `from datasette.utils import path_with_format` in https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/__init__.py just now. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
779691739 | |
https://github.com/simonw/datasette/issues/1178#issuecomment-755158310 | https://api.github.com/repos/simonw/datasette/issues/1178 | 755158310 | MDEyOklzc3VlQ29tbWVudDc1NTE1ODMxMA== | 9599 | 2021-01-06T08:25:31Z | 2021-01-06T08:25:31Z | OWNER | Moving this to the Datasette repo. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780267857 | |
https://github.com/simonw/datasette/issues/1178#issuecomment-755157732 | https://api.github.com/repos/simonw/datasette/issues/1178 | 755157732 | MDEyOklzc3VlQ29tbWVudDc1NTE1NzczMg== | 9599 | 2021-01-06T08:24:12Z | 2021-01-06T08:24:12Z | OWNER | https://latest-with-plugins.datasette.io/fixtures/sortable.json has the bug too - the `next_url` is `http://` when it should be `https://`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780267857 | |
https://github.com/simonw/datasette/issues/1178#issuecomment-755157281 | https://api.github.com/repos/simonw/datasette/issues/1178 | 755157281 | MDEyOklzc3VlQ29tbWVudDc1NTE1NzI4MQ== | 9599 | 2021-01-06T08:23:14Z | 2021-01-06T08:23:14Z | OWNER | https://latest-with-plugins.datasette.io/-/settings says `"force_https_urls": false` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780267857 | |
https://github.com/simonw/datasette/issues/1178#issuecomment-755157066 | https://api.github.com/repos/simonw/datasette/issues/1178 | 755157066 | MDEyOklzc3VlQ29tbWVudDc1NTE1NzA2Ng== | 9599 | 2021-01-06T08:22:47Z | 2021-01-06T08:22:47Z | OWNER | Weird... https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/app.py#L609-L613 ```python def absolute_url(self, request, path): url = urllib.parse.urljoin(request.url, path) if url.startswith("http://") and self.setting("force_https_urls"): url = "https://" + url[len("http://") :] return url ``` That looks like it should work. Needs more digging. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780267857 | |
https://github.com/simonw/datasette/issues/1178#issuecomment-755156606 | https://api.github.com/repos/simonw/datasette/issues/1178 | 755156606 | MDEyOklzc3VlQ29tbWVudDc1NTE1NjYwNg== | 9599 | 2021-01-06T08:21:49Z | 2021-01-06T08:21:49Z | OWNER | https://github.com/simonw/datasette-export-notebook/blob/aec398eab4f34791d240d7bc47b6eec575b357be/datasette_export_notebook/__init__.py#L18-L23 Maybe this is a bug in `datasette.absolute_url`? Perhaps it doesn't take the scheme into account. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
780267857 | |
https://github.com/simonw/datasette/issues/1101#issuecomment-755134771 | https://api.github.com/repos/simonw/datasette/issues/1101 | 755134771 | MDEyOklzc3VlQ29tbWVudDc1NTEzNDc3MQ== | 9599 | 2021-01-06T07:28:01Z | 2021-01-06T07:28:01Z | OWNER | With this structure it will become possible to stream non-newline-delimited JSON array-of-objects too - the `stream_rows()` method could output `[` first, then each row followed by a comma, then `]` after the very last row. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
749283032 | |
https://github.com/simonw/datasette/issues/1101#issuecomment-755133937 | https://api.github.com/repos/simonw/datasette/issues/1101 | 755133937 | MDEyOklzc3VlQ29tbWVudDc1NTEzMzkzNw== | 9599 | 2021-01-06T07:25:48Z | 2021-01-06T07:26:43Z | OWNER | Idea: instead of returning a dictionary, `register_output_renderer` could return an object. The object could have the following properties: - `.extension` - the extension to use - `.can_render(...)` - says if it can render this - `.can_stream(...)` - says if streaming is supported - `async .stream_rows(rows_iterator, send)` - method that loops through all rows and uses `send` to send them to the response in the correct format I can then deprecate the existing `dict` return type for 1.0. | { "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
749283032 | |
https://github.com/simonw/datasette/issues/1101#issuecomment-755128038 | https://api.github.com/repos/simonw/datasette/issues/1101 | 755128038 | MDEyOklzc3VlQ29tbWVudDc1NTEyODAzOA== | 9599 | 2021-01-06T07:10:22Z | 2021-01-06T07:10:22Z | OWNER | Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas: pandas.read_json("https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on", lines=True) | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
749283032 | |
https://github.com/simonw/datasette/issues/1171#issuecomment-754958998 | https://api.github.com/repos/simonw/datasette/issues/1171 | 754958998 | MDEyOklzc3VlQ29tbWVudDc1NDk1ODk5OA== | 9599 | 2021-01-05T23:16:33Z | 2021-01-05T23:16:33Z | OWNER | That's really useful, thanks @rcoup | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
778450486 | |
https://github.com/simonw/datasette/issues/782#issuecomment-754958610 | https://api.github.com/repos/simonw/datasette/issues/782 | 754958610 | MDEyOklzc3VlQ29tbWVudDc1NDk1ODYxMA== | 9599 | 2021-01-05T23:15:24Z | 2021-01-05T23:15:24Z | OWNER | https://latest-with-plugins.datasette.io/fixtures/roadside_attraction_characteristics/1.json-preview returns a 500 error at the moment - a KeyError on 'filtered_table_rows_count'. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
627794879 | |
https://github.com/simonw/datasette/issues/576#issuecomment-754957658 | https://api.github.com/repos/simonw/datasette/issues/576 | 754957658 | MDEyOklzc3VlQ29tbWVudDc1NDk1NzY1OA== | 9599 | 2021-01-05T23:12:50Z | 2021-01-05T23:12:50Z | OWNER | See https://docs.datasette.io/en/stable/internals.html | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
497170355 | |
https://github.com/simonw/datasette/issues/576#issuecomment-754957563 | https://api.github.com/repos/simonw/datasette/issues/576 | 754957563 | MDEyOklzc3VlQ29tbWVudDc1NDk1NzU2Mw== | 9599 | 2021-01-05T23:12:37Z | 2021-01-05T23:12:37Z | OWNER | I'm happy with how this has evolved, so I'm closing the issue. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
497170355 | |
https://github.com/simonw/datasette/issues/1176#issuecomment-754957378 | https://api.github.com/repos/simonw/datasette/issues/1176 | 754957378 | MDEyOklzc3VlQ29tbWVudDc1NDk1NzM3OA== | 9599 | 2021-01-05T23:12:03Z | 2021-01-05T23:12:03Z | OWNER | This needs to be done for Datasette 1.0. At the very least I need to ensure it's clear that `datasette.utils` is not part of the public API unless explicitly marked as such. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
779691739 | |
https://github.com/simonw/datasette/issues/1176#issuecomment-754952146 | https://api.github.com/repos/simonw/datasette/issues/1176 | 754952146 | MDEyOklzc3VlQ29tbWVudDc1NDk1MjE0Ng== | 9599 | 2021-01-05T22:57:26Z | 2021-01-05T22:57:26Z | OWNER | Known public APIs might be worth adding type annotations to as well. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
779691739 | |
https://github.com/simonw/datasette/issues/1176#issuecomment-754952040 | https://api.github.com/repos/simonw/datasette/issues/1176 | 754952040 | MDEyOklzc3VlQ29tbWVudDc1NDk1MjA0MA== | 9599 | 2021-01-05T22:57:09Z | 2021-01-05T22:57:09Z | OWNER | It might be neater to move all of the non-public functions into a separate module - `datasette.utils.internal` perhaps. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
779691739 | |
https://github.com/simonw/datasette/issues/1176#issuecomment-754951786 | https://api.github.com/repos/simonw/datasette/issues/1176 | 754951786 | MDEyOklzc3VlQ29tbWVudDc1NDk1MTc4Ng== | 9599 | 2021-01-05T22:56:27Z | 2021-01-05T22:56:43Z | OWNER | Idea: introduce a `@documented` decorator which marks specific functions as part of the public, documented API. The unit tests can then confirm that anything with that decorator is both documented and tested. ```python @documented def escape_css_string(s): return _css_re.sub( lambda m: "\\" + (f"{ord(m.group()):X}".zfill(6)), s.replace("\r\n", "\n"), ) ``` Or maybe `@public`? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
779691739 | |
https://github.com/simonw/datasette/issues/1175#issuecomment-754696725 | https://api.github.com/repos/simonw/datasette/issues/1175 | 754696725 | MDEyOklzc3VlQ29tbWVudDc1NDY5NjcyNQ== | 9599 | 2021-01-05T15:12:30Z | 2021-01-05T15:12:30Z | OWNER | Some tips here: https://github.com/tiangolo/fastapi/issues/78 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
779156520 | |
https://github.com/simonw/datasette/issues/1173#issuecomment-754463845 | https://api.github.com/repos/simonw/datasette/issues/1173 | 754463845 | MDEyOklzc3VlQ29tbWVudDc1NDQ2Mzg0NQ== | 9599 | 2021-01-05T07:41:43Z | 2021-01-05T07:41:43Z | OWNER | https://github.com/oleksis/pyinstaller-manylinux looks useful, via https://twitter.com/oleksis/status/1346341987876823040 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
778682317 | |
https://github.com/simonw/datasette/issues/1171#issuecomment-754296761 | https://api.github.com/repos/simonw/datasette/issues/1171 | 754296761 | MDEyOklzc3VlQ29tbWVudDc1NDI5Njc2MQ== | 9599 | 2021-01-04T23:55:44Z | 2021-01-04T23:55:44Z | OWNER | Bit uncomfortable that it looks like you need to include your Apple ID username and password in the CI configuration to do this. I'll use GitHub Secrets for this but I don't like it - I'll definitely setup a dedicated code signing account that's not my access-to-everything AppleID for this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
778450486 | |
https://github.com/simonw/datasette/issues/1171#issuecomment-754295380 | https://api.github.com/repos/simonw/datasette/issues/1171 | 754295380 | MDEyOklzc3VlQ29tbWVudDc1NDI5NTM4MA== | 9599 | 2021-01-04T23:54:32Z | 2021-01-04T23:54:32Z | OWNER | https://github.com/search?l=YAML&q=gon+json&type=Code reveals some examples of people using `gon` in workflows. These look useful: * https://github.com/coherence/hub-server/blob/3b7e9c7c5bce9e244b14b854f1f89d66f53a5a39/.github/workflows/release_build.yml * https://github.com/simoncozens/pilcrow/blob/5abc145e7fb9577086afe47b48fd730cb8195386/.github/workflows/buildapp.yaml | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
778450486 | |
https://github.com/simonw/datasette/issues/1171#issuecomment-754287882 | https://api.github.com/repos/simonw/datasette/issues/1171 | 754287882 | MDEyOklzc3VlQ29tbWVudDc1NDI4Nzg4Mg== | 9599 | 2021-01-04T23:40:10Z | 2021-01-04T23:42:32Z | OWNER | This looks VERY useful: https://github.com/mitchellh/gon - " Sign, notarize, and package macOS CLI tools and applications written in any language. Available as both a CLI and a Go library." And it installs like this: brew install mitchellh/gon/gon | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
778450486 | |
https://github.com/simonw/datasette/issues/1171#issuecomment-754286783 | https://api.github.com/repos/simonw/datasette/issues/1171 | 754286783 | MDEyOklzc3VlQ29tbWVudDc1NDI4Njc4Mw== | 9599 | 2021-01-04T23:38:18Z | 2021-01-04T23:38:18Z | OWNER | Oh wow maybe I need to Notarize it too? https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
778450486 | |
https://github.com/simonw/datasette/issues/1171#issuecomment-754286618 | https://api.github.com/repos/simonw/datasette/issues/1171 | 754286618 | MDEyOklzc3VlQ29tbWVudDc1NDI4NjYxOA== | 9599 | 2021-01-04T23:37:45Z | 2021-01-04T23:37:45Z | OWNER | https://github.com/actions/virtual-environments/issues/1820#issuecomment-719549887 looks useful - not sure if those notes are for iOS or macOS though. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
778450486 | |
https://github.com/simonw/datasette/issues/93#issuecomment-754285795 | https://api.github.com/repos/simonw/datasette/issues/93 | 754285795 | MDEyOklzc3VlQ29tbWVudDc1NDI4NTc5NQ== | 9599 | 2021-01-04T23:35:13Z | 2021-01-04T23:35:13Z | OWNER | Next step is to automate this all! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
273944952 | |
https://github.com/simonw/datasette/issues/1152#issuecomment-754285588 | https://api.github.com/repos/simonw/datasette/issues/1152 | 754285588 | MDEyOklzc3VlQ29tbWVudDc1NDI4NTU4OA== | 9599 | 2021-01-04T23:34:30Z | 2021-01-04T23:34:30Z | OWNER | I think the way to do this is to have a new plugin hook that returns two SQL where clauses: one returning a list of resources that the user should be able to access (the allow-list) and one returning a list of resources they are explicitly forbidden from accessing (the deny-list). Either of these can be blank. Datasette can then combine those into a full SQL query and use it to answer the question "show me a list of resources that the user is allowed to perform action X on". It can also answer the existing question, "is user X allowed to perform action Y on resource Z"? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
770598024 | |
https://github.com/simonw/datasette/issues/93#issuecomment-754233960 | https://api.github.com/repos/simonw/datasette/issues/93 | 754233960 | MDEyOklzc3VlQ29tbWVudDc1NDIzMzk2MA== | 9599 | 2021-01-04T21:35:37Z | 2021-01-04T21:35:37Z | OWNER | I tested it by running a `tmate` session on the GitHub macOS machines, and it worked! ``` Mac-1609795972770:tmp runner$ wget 'https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip' --2021-01-04 21:34:10-- https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip Resolving github.com (github.com)... 140.82.114.4 Connecting to github.com (github.com)|140.82.114.4|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream [following] --2021-01-04 21:34:14-- https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.43.164 Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.43.164|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 8297283 (7.9M) [applicat… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
273944952 | |
https://github.com/simonw/datasette/issues/93#issuecomment-754229977 | https://api.github.com/repos/simonw/datasette/issues/93 | 754229977 | MDEyOklzc3VlQ29tbWVudDc1NDIyOTk3Nw== | 9599 | 2021-01-04T21:28:01Z | 2021-01-04T21:28:01Z | OWNER | As an experiment, I put the macOS one in a zip file and attached it to the latest release: ``` mkdir datasette-0.53-macos-binary cp dist/datasette datasette-0.53-macos-binary zip -r datasette-0.53-macos-binary.zip datasette-0.53-macos-binary ``` It's available here: https://github.com/simonw/datasette/releases/tag/0.53 - download URL is https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
273944952 | |
https://github.com/simonw/datasette/issues/93#issuecomment-754227543 | https://api.github.com/repos/simonw/datasette/issues/93 | 754227543 | MDEyOklzc3VlQ29tbWVudDc1NDIyNzU0Mw== | 9599 | 2021-01-04T21:23:13Z | 2021-01-04T21:23:13Z | OWNER | ``` (pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# dist/datasette --get /-/databases.json [{"name": ":memory:", "path": null, "size": 0, "is_mutable": true, "is_memory": true, "hash": null}] (pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# ls -lah dist/datasette -rwxr-xr-x 1 root root 8.9M Jan 4 21:05 dist/datasette ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
273944952 | |
https://github.com/simonw/datasette/issues/93#issuecomment-754219002 | https://api.github.com/repos/simonw/datasette/issues/93 | 754219002 | MDEyOklzc3VlQ29tbWVudDc1NDIxOTAwMg== | 9599 | 2021-01-04T21:06:49Z | 2021-01-04T21:22:27Z | OWNER | Works on Linux/Ubuntu too, except I had to do `export BASE=` on a separate line. I also did this: ``` apt-get install python3 python3-venv python3 -m venv pyinstaller-venv source pyinstaller-venv/bin/activate pip install wheel pip install datasette pyinstaller export DATASETTE_BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))') pyinstaller -F \ --add-data "$DATASETTE_BASE/templates:datasette/templates" \ --add-data "$DATASETTE_BASE/static:datasette/static" \ --hidden-import datasette.publish \ --hidden-import datasette.publish.heroku \ --hidden-import datasette.publish.cloudrun \ --hidden-import datasette.facets \ --hidden-import datasette.sql_functions \ --hidden-import datasette.actor_auth_cookie \ --hidden-import datasette.default_permissions \ --hidden-import datasette.default_magic_parameters \ --hidden-import datasette.blob_renderer \ --hidden-import datasette.default_menu_links \ --hidden-import uvicorn \ --hidden-import uvicorn.logging \ --hidden-import uvicorn.loops \ --hidden-import uvicorn.loops.auto \ --hidden-import uvicorn.protocols \ --hidden-import uvicorn.protocols.http \ --hidden-import uvicorn.protocols.http.auto \ --hidden-import uvicorn.protocols.websockets \ --hidden-import uvicorn.protocols.websockets.auto \ --hidden-import uvicorn.lifespan \ --hidden-import uvicorn.lifespan.on \ $(which datasette) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
273944952 | |
https://github.com/simonw/datasette/issues/93#issuecomment-754218545 | https://api.github.com/repos/simonw/datasette/issues/93 | 754218545 | MDEyOklzc3VlQ29tbWVudDc1NDIxODU0NQ== | 9599 | 2021-01-04T21:05:57Z | 2021-01-04T21:05:57Z | OWNER | That BASE= trick seems to work with `zsh` but not with `bash`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
273944952 | |
https://github.com/simonw/datasette/issues/93#issuecomment-754215392 | https://api.github.com/repos/simonw/datasette/issues/93 | 754215392 | MDEyOklzc3VlQ29tbWVudDc1NDIxNTM5Mg== | 9599 | 2021-01-04T20:59:20Z | 2021-01-04T21:03:14Z | OWNER | Updated `pyinstaller` recipe - lots of hidden imports needed now: ``` pip install wheel pip install datasette pyinstaller BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))') \ pyinstaller -F \ --add-data "$BASE/templates:datasette/templates" \ --add-data "$BASE/static:datasette/static" \ --hidden-import datasette.publish \ --hidden-import datasette.publish.heroku \ --hidden-import datasette.publish.cloudrun \ --hidden-import datasette.facets \ --hidden-import datasette.sql_functions \ --hidden-import datasette.actor_auth_cookie \ --hidden-import datasette.default_permissions \ --hidden-import datasette.default_magic_parameters \ --hidden-import datasette.blob_renderer \ --hidden-import datasette.default_menu_links \ --hidden-import uvicorn \ --hidden-import uvicorn.logging \ --hidden-import uvicorn.loops \ --hidden-import uvicorn.loops.auto \ --hidden-import uvicorn.protocols \ --hidden-import uvicorn.protocols.http \ --hidden-import uvicorn.protocols.http.auto \ --hidden-import uvicorn.protocols.websockets \ --hidden-import uvicorn.protocols.websockets.auto \ --hidden-import uvicorn.lifespan \ --hidden-import uvicorn.lifespan.on \ $(which datasette) ``` | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
273944952 | |
https://github.com/simonw/datasette/issues/93#issuecomment-754215793 | https://api.github.com/repos/simonw/datasette/issues/93 | 754215793 | MDEyOklzc3VlQ29tbWVudDc1NDIxNTc5Mw== | 9599 | 2021-01-04T21:00:14Z | 2021-01-04T21:00:14Z | OWNER | ``` (pyinstaller-datasette) pyinstaller-datasette % file dist/datasette dist/datasette: Mach-O 64-bit executable x86_64 (pyinstaller-datasette) pyinstaller-datasette % ls -lah dist/datasette -rwxr-xr-x 1 simon wheel 8.0M Jan 4 12:58 dist/datasette ``` I'm surprised it's only 8MB! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
273944952 | |
https://github.com/simonw/datasette/issues/668#issuecomment-754194996 | https://api.github.com/repos/simonw/datasette/issues/668 | 754194996 | MDEyOklzc3VlQ29tbWVudDc1NDE5NDk5Ng== | 9599 | 2021-01-04T20:18:39Z | 2021-01-04T20:18:39Z | OWNER | I fixed this in #1115 - you can run `--load-extension=spatialite` now and it will look for the extension in common places. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
563347679 | |
https://github.com/simonw/datasette/issues/436#issuecomment-754193501 | https://api.github.com/repos/simonw/datasette/issues/436 | 754193501 | MDEyOklzc3VlQ29tbWVudDc1NDE5MzUwMQ== | 9599 | 2021-01-04T20:15:41Z | 2021-01-04T20:15:41Z | OWNER | Sadly `publish.datasettes.com` was broken by changes to Zeit, and I don't think I'll be bringing it back. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
435819321 | |
https://github.com/simonw/datasette/issues/371#issuecomment-754192873 | https://api.github.com/repos/simonw/datasette/issues/371 | 754192873 | MDEyOklzc3VlQ29tbWVudDc1NDE5Mjg3Mw== | 9599 | 2021-01-04T20:14:28Z | 2021-01-04T20:14:28Z | OWNER | Now that Digital Ocean has App Platform this is less necessary, especially since the documentation covers how to use App Platform here: https://docs.datasette.io/en/stable/deploying.html#deploying-using-buildpacks | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
377156339 | |
https://github.com/simonw/datasette/issues/102#issuecomment-754192267 | https://api.github.com/repos/simonw/datasette/issues/102 | 754192267 | MDEyOklzc3VlQ29tbWVudDc1NDE5MjI2Nw== | 9599 | 2021-01-04T20:13:19Z | 2021-01-04T20:13:19Z | OWNER | I'm more likely to do Lambda than Elastic Beanstalk, especially now the size limit for Lambdas has been increased as part of their support for Docker. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
274264175 | |
https://github.com/simonw/datasette/issues/221#issuecomment-754191699 | https://api.github.com/repos/simonw/datasette/issues/221 | 754191699 | MDEyOklzc3VlQ29tbWVudDc1NDE5MTY5OQ== | 9599 | 2021-01-04T20:12:14Z | 2021-01-04T20:12:14Z | OWNER | I'm going to close this. Plugins can register their own CLI tools (see https://github.com/simonw/click-app) if they need to. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
315142414 | |
https://github.com/simonw/datasette/issues/221#issuecomment-754190952 | https://api.github.com/repos/simonw/datasette/issues/221 | 754190952 | MDEyOklzc3VlQ29tbWVudDc1NDE5MDk1Mg== | 9599 | 2021-01-04T20:10:51Z | 2021-01-04T20:10:51Z | OWNER | Is this still a good idea? I don't have any pressing need for it at the moment. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
315142414 | |
https://github.com/simonw/datasette/issues/221#issuecomment-754190814 | https://api.github.com/repos/simonw/datasette/issues/221 | 754190814 | MDEyOklzc3VlQ29tbWVudDc1NDE5MDgxNA== | 9599 | 2021-01-04T20:10:34Z | 2021-01-04T20:10:34Z | OWNER | For the `csvs-to-sqlite` case I'm going with `datasette insert` instead, see #1160. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
315142414 | |
https://github.com/simonw/datasette/issues/18#issuecomment-754188383 | https://api.github.com/repos/simonw/datasette/issues/18 | 754188383 | MDEyOklzc3VlQ29tbWVudDc1NDE4ODM4Mw== | 9599 | 2021-01-04T20:05:48Z | 2021-01-04T20:05:48Z | OWNER | I'm not using Sanic any more, but this is still very feasible. If I ever do it I'll write a plugin. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
267739593 | |
https://github.com/simonw/datasette/issues/103#issuecomment-754188099 | https://api.github.com/repos/simonw/datasette/issues/103 | 754188099 | MDEyOklzc3VlQ29tbWVudDc1NDE4ODA5OQ== | 9599 | 2021-01-04T20:05:14Z | 2021-01-04T20:05:14Z | OWNER | Wontfix, Cloud Run is already implemented and is a better fit for Datasette. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
274265878 | |
https://github.com/simonw/datasette/issues/913#issuecomment-754187520 | https://api.github.com/repos/simonw/datasette/issues/913 | 754187520 | MDEyOklzc3VlQ29tbWVudDc1NDE4NzUyMA== | 9599 | 2021-01-04T20:04:10Z | 2021-01-04T20:04:10Z | OWNER | That's pretty elegant: each plugin gets its own namespace and can register new settings. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
670209331 | |
https://github.com/simonw/datasette/issues/913#issuecomment-754187326 | https://api.github.com/repos/simonw/datasette/issues/913 | 754187326 | MDEyOklzc3VlQ29tbWVudDc1NDE4NzMyNg== | 9599 | 2021-01-04T20:03:50Z | 2021-01-04T20:03:50Z | OWNER | I renamed `--config` to `--setting` and changed it to work like this: datasette --setting sql_time_limit_ms 1000 Note the lack of colons. This actually makes colons cleaner to use for plugins - I could support this: datasette --setting datasette-insert:unsafe 1 | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
670209331 | |
https://github.com/simonw/datasette/issues/1111#issuecomment-754184287 | https://api.github.com/repos/simonw/datasette/issues/1111 | 754184287 | MDEyOklzc3VlQ29tbWVudDc1NDE4NDI4Nw== | 9599 | 2021-01-04T19:57:53Z | 2021-01-04T19:57:53Z | OWNER | Relevant new feature in sqlite-utils: the ability to use triggers to maintain fast counts. This optimization could help a lot here. https://github.com/simonw/sqlite-utils/issues/212 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
751195017 | |
https://github.com/simonw/datasette/issues/1164#issuecomment-754182058 | https://api.github.com/repos/simonw/datasette/issues/1164 | 754182058 | MDEyOklzc3VlQ29tbWVudDc1NDE4MjA1OA== | 9599 | 2021-01-04T19:53:31Z | 2021-01-04T19:53:31Z | OWNER | This will be helped by the new `package.json` added in #1170. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
776634318 | |
https://github.com/simonw/datasette/pull/1170#issuecomment-754181646 | https://api.github.com/repos/simonw/datasette/issues/1170 | 754181646 | MDEyOklzc3VlQ29tbWVudDc1NDE4MTY0Ng== | 9599 | 2021-01-04T19:52:40Z | 2021-01-04T19:52:40Z | OWNER | Thank you very much! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
778126516 | |
https://github.com/simonw/datasette/issues/983#issuecomment-753690280 | https://api.github.com/repos/simonw/datasette/issues/983 | 753690280 | MDEyOklzc3VlQ29tbWVudDc1MzY5MDI4MA== | 9599 | 2021-01-03T23:13:30Z | 2021-01-03T23:13:30Z | OWNER | Oh that's interesting, I hadn't thought about plugins firing events - just responding to events fired by the rest of the application. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
712260429 | |
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753671902 | https://api.github.com/repos/simonw/sqlite-utils/issues/219 | 753671902 | MDEyOklzc3VlQ29tbWVudDc1MzY3MTkwMg== | 9599 | 2021-01-03T20:31:04Z | 2021-01-03T20:32:13Z | OWNER | A `table.has_count_triggers` property. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777707544 | |
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753671235 | https://api.github.com/repos/simonw/sqlite-utils/issues/219 | 753671235 | MDEyOklzc3VlQ29tbWVudDc1MzY3MTIzNQ== | 9599 | 2021-01-03T20:25:10Z | 2021-01-03T20:25:10Z | OWNER | To detect tables, look at the names of the triggers - `{table}{counts_table}_insert` and `{table}{counts_table}_delete`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777707544 | |
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753671009 | https://api.github.com/repos/simonw/sqlite-utils/issues/219 | 753671009 | MDEyOklzc3VlQ29tbWVudDc1MzY3MTAwOQ== | 9599 | 2021-01-03T20:22:53Z | 2021-01-03T20:22:53Z | OWNER | I think this should be accompanied by a `sqlite-utils reset-counts` command. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777707544 | |
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753670833 | https://api.github.com/repos/simonw/sqlite-utils/issues/219 | 753670833 | MDEyOklzc3VlQ29tbWVudDc1MzY3MDgzMw== | 9599 | 2021-01-03T20:20:54Z | 2021-01-03T20:20:54Z | OWNER | This is a little tricky. We should assume that the existing values in the `_counts` table cannot be trusted at all when this method is called - so we should probably clear that table entirely and then re-populate it. But that means we need to figure out which tables in the database have the counts triggers defined. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777707544 | |
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753668099 | https://api.github.com/repos/simonw/sqlite-utils/issues/215 | 753668099 | MDEyOklzc3VlQ29tbWVudDc1MzY2ODA5OQ== | 9599 | 2021-01-03T19:55:53Z | 2021-01-03T19:55:53Z | OWNER | So if you instantiate the `Database()` constructor with `use_counts_table=True` any access to the `.count` properties will go through this table - otherwise regular `count(*)` queries will be executed. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777535402 | |
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753665521 | https://api.github.com/repos/simonw/sqlite-utils/issues/215 | 753665521 | MDEyOklzc3VlQ29tbWVudDc1MzY2NTUyMQ== | 9599 | 2021-01-03T19:31:33Z | 2021-01-03T19:31:33Z | OWNER | I'm having second thoughts about this being the default behaviour. It's pretty weird. I feel like HUGE databases that need this are rare, so having it on by default doesn't make sense. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777535402 | |
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753662490 | https://api.github.com/repos/simonw/sqlite-utils/issues/215 | 753662490 | MDEyOklzc3VlQ29tbWVudDc1MzY2MjQ5MA== | 9599 | 2021-01-03T19:05:53Z | 2021-01-03T19:05:53Z | OWNER | Idea: a `.execute_count()` method that never uses the cache. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777535402 | |
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753661292 | https://api.github.com/repos/simonw/sqlite-utils/issues/215 | 753661292 | MDEyOklzc3VlQ29tbWVudDc1MzY2MTI5Mg== | 9599 | 2021-01-03T18:56:06Z | 2021-01-03T18:56:23Z | OWNER | Another option: on creation of the `Database()` object, check to see if the `_counts` table exists and use that as the default for a `use_counts_table` property. Also flip that property to `True` if the user calls `.enable_counts()` at any time. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777535402 | |
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753661158 | https://api.github.com/repos/simonw/sqlite-utils/issues/215 | 753661158 | MDEyOklzc3VlQ29tbWVudDc1MzY2MTE1OA== | 9599 | 2021-01-03T18:55:16Z | 2021-01-03T18:55:16Z | OWNER | Alternative implementation: provided `db.should_trust_counts` is `True`, try running the query: ```sql select count from _counts where [table] = ? ``` If the query fails to return a result OR throws an error because the table doesn't exist, run the `count(*)` query. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777535402 | |
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753660814 | https://api.github.com/repos/simonw/sqlite-utils/issues/215 | 753660814 | MDEyOklzc3VlQ29tbWVudDc1MzY2MDgxNA== | 9599 | 2021-01-03T18:53:05Z | 2021-01-03T18:53:05Z | OWNER | Here's the current `.count` property: https://github.com/simonw/sqlite-utils/blob/036ec6d32313487527c66dea613a3e7118b97459/sqlite_utils/db.py#L597-L609 It's implemented on `Queryable` which means it's available on both `Table` and `View` - the optimization doesn't make sense for views. I'm a bit cautious about making that property so much more complex. In order to decide if it should try the `_counts` table first it needs to know: - Should it be trusting the counts? I'm thinking a `.should_trust_counts` property on `Database` which defaults to `True` would be good - then advanced users can turn that off if they know the counts should not be trusted. - Does the `_counts` table exist? - Are the triggers defined? Then it can do the query, and if the query fails it can fall back on the `count(*)`. That's quite a lot of extra activity though. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777535402 | |
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753660379 | https://api.github.com/repos/simonw/sqlite-utils/issues/215 | 753660379 | MDEyOklzc3VlQ29tbWVudDc1MzY2MDM3OQ== | 9599 | 2021-01-03T18:50:15Z | 2021-01-03T18:50:15Z | OWNER | ```python def cached_counts(self, tables=None): sql = "select [table], count from {}".format(self._counts_table_name) if tables: sql += " where [table] in ({})".format(", ".join("?" for table in tables)) return {r[0]: r[1] for r in self.execute(sql, tables).fetchall()} ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777535402 | |
https://github.com/simonw/sqlite-utils/issues/206#issuecomment-753659260 | https://api.github.com/repos/simonw/sqlite-utils/issues/206 | 753659260 | MDEyOklzc3VlQ29tbWVudDc1MzY1OTI2MA== | 9599 | 2021-01-03T18:42:01Z | 2021-01-03T18:42:01Z | OWNER | ``` % sqlite-utils insert blah.db blah global_power_plant_database.csv Error: Invalid JSON - use --csv for CSV or --tsv for TSV files ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
761915790 | |
https://github.com/simonw/datasette/issues/1169#issuecomment-753657180 | https://api.github.com/repos/simonw/datasette/issues/1169 | 753657180 | MDEyOklzc3VlQ29tbWVudDc1MzY1NzE4MA== | 9599 | 2021-01-03T18:23:30Z | 2021-01-03T18:23:30Z | OWNER | Also welcome in that PR would be a bit of documentation for contributors, see #1167 - but no problem if you leave that out, I'm happy to add it later. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777677671 | |
https://github.com/simonw/datasette/issues/1169#issuecomment-753653260 | https://api.github.com/repos/simonw/datasette/issues/1169 | 753653260 | MDEyOklzc3VlQ29tbWVudDc1MzY1MzI2MA== | 9599 | 2021-01-03T17:54:40Z | 2021-01-03T17:54:40Z | OWNER | And @benpickles yes I would land that pull request straight away as-is. Thanks! | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777677671 | |
https://github.com/simonw/datasette/issues/1169#issuecomment-753653033 | https://api.github.com/repos/simonw/datasette/issues/1169 | 753653033 | MDEyOklzc3VlQ29tbWVudDc1MzY1MzAzMw== | 9599 | 2021-01-03T17:52:53Z | 2021-01-03T17:52:53Z | OWNER | Oh that's so frustrating! I was worried about that - I spotted a few runs that seemed faster and hoped that it meant that the package was coming out of the `~/.npm` cache, but evidently that's not the case. You've convinced me that Datasette itself should have a `package.json` - the Dependabot argument is a really good one. But... I'd really love to figure out a general pattern for using `npx` scripts in GitHub Actions workflows in a cache-friendly way. I have plenty of other projects that I'd love to run Prettier or Uglify or `puppeteer-cli` in without adding a `package.json` to them. Any ideas? The best I can think of is for the workflow itself to write out a `package.json` file (using `echo '{ ... }' > package.json`) as part of the run - that way the cache should work (I think) but I don't get a misleading `package.json` file sitting in the repo. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777677671 | |
https://github.com/simonw/datasette/issues/983#issuecomment-753570710 | https://api.github.com/repos/simonw/datasette/issues/983 | 753570710 | MDEyOklzc3VlQ29tbWVudDc1MzU3MDcxMA== | 9599 | 2021-01-03T05:29:56Z | 2021-01-03T05:29:56Z | OWNER | I thought about using browser events, but they don't quite match the API that I'm looking to provide. In particular, the great thing about Pluggy is that if you have multiple handlers registered for a specific plugin hook each of those handlers can return a value, and Pluggy will combine those values into a list of replies. This is great for things like plugin hooks that add extra menu items - each plugin can return a menu item (maybe as a label/URL/click-callback object) and the calling code can then add all of those items to the menu. See https://docs.datasette.io/en/stable/plugin_hooks.html#table-actions-datasette-actor-database-table for a Python example. I'm on the fence about relying on JavaScript modules. I need to think about browser compatibility for them - but I'm already commited to requiring support for `() => {}` arrow functions so maybe I'm committed to module support too already? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
712260429 | |
https://github.com/simonw/datasette/issues/1160#issuecomment-753568428 | https://api.github.com/repos/simonw/datasette/issues/1160 | 753568428 | MDEyOklzc3VlQ29tbWVudDc1MzU2ODQyOA== | 9599 | 2021-01-03T05:02:32Z | 2021-01-03T05:02:32Z | OWNER | Should this command include a `--fts` option for configuring full-text search on one-or-more columns? I thought about doing that for `sqlite-utils insert` in https://github.com/simonw/sqlite-utils/issues/202 and decided not to because of the need to include extra options covering the FTS version, porter stemming options and whether or not to create triggers. But maybe I can set sensible defaults for that with `datasette insert ... -f title -f body`? Worth thinking about a bit more. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
775666296 | |
https://github.com/simonw/sqlite-utils/issues/202#issuecomment-753568264 | https://api.github.com/repos/simonw/sqlite-utils/issues/202 | 753568264 | MDEyOklzc3VlQ29tbWVudDc1MzU2ODI2NA== | 9599 | 2021-01-03T05:00:24Z | 2021-01-03T05:00:24Z | OWNER | I'm not going to implement this, because it actually needs several additional options that already exist on `sqlite-utils enable-fts`: ``` --fts4 Use FTS4 --fts5 Use FTS5 --tokenize TEXT Tokenizer to use, e.g. porter --create-triggers Create triggers to update the FTS tables when the parent table changes. ``` I'd rather not add all four of those options to `sqlite-utils insert` just to support this shortcut. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
738514367 | |
https://github.com/simonw/sqlite-utils/issues/202#issuecomment-753567969 | https://api.github.com/repos/simonw/sqlite-utils/issues/202 | 753567969 | MDEyOklzc3VlQ29tbWVudDc1MzU2Nzk2OQ== | 9599 | 2021-01-03T04:55:17Z | 2021-01-03T04:55:43Z | OWNER | The long version of this can be `--fts`, same as in `csvs-to-sqlite`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
738514367 | |
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932 | https://api.github.com/repos/simonw/sqlite-utils/issues/203 | 753567932 | MDEyOklzc3VlQ29tbWVudDc1MzU2NzkzMg== | 9599 | 2021-01-03T04:54:43Z | 2021-01-03T04:54:43Z | OWNER | Another option: expand the `ForeignKey` object to have `.columns` and `.other_columns` properties in addition to the existing `.column` and `.other_column` properties. These new plural properties would always return a tuple, which would be a one-item tuple for a non-compound-foreign-key. The question then is what should `.column` and `.other_column` return for compound foreign keys? I'd be inclined to say they should return `None` - which would trigger errors in code that encounters a compound foreign key for the first time, but those errors would at least be a strong indicator as to what had gone wrong. We can label `.column` and `.other_column` as deprecated and then remove them in `sqlite-utils 4.0`. Since this would still be a breaking change in some minor edge-cases I'm thinking maybe 4.0 needs to happen in order to land this feature. I'm not opposed to doing that, I was just hoping it might be avoidable. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
743384829 | |
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567744 | https://api.github.com/repos/simonw/sqlite-utils/issues/203 | 753567744 | MDEyOklzc3VlQ29tbWVudDc1MzU2Nzc0NA== | 9599 | 2021-01-03T04:51:44Z | 2021-01-03T04:51:44Z | OWNER | One way that this could avoid a breaking change would be to have `fk.column` and `fk.other_column` remain as strings for non-compound-foreign-keys, but turn into tuples for a compound foreign key. This is a bit of an ugly API design, and it could still break existing code that encounters a compound foreign key for the first time - but it would leave code working for the more common case of a non-compound-foreign-key. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
743384829 | |
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567508 | https://api.github.com/repos/simonw/sqlite-utils/issues/203 | 753567508 | MDEyOklzc3VlQ29tbWVudDc1MzU2NzUwOA== | 9599 | 2021-01-03T04:48:17Z | 2021-01-03T04:48:17Z | OWNER | Sorry for taking so long to review this! This approach looks great to me - being able to optionally pass a tuple anywhere the API currently expects a column is smart, and it's consistent with how the `pk=` parameter works elsewhere. There's just one problem I can see with this: the way it changes the `ForeignKey(...)` interface to always return a tuple for `.column` and `.other_column`, even if that tuple only contains a single item. This represents a breaking change to the existing API - any code that expects `ForeignKey.column` to be a single string (which is any code that has been written against that) will break. As such, I'd have to bump the major version of `sqlite-utils` to `4.0` in order to ship this. Ideally I'd like to make this change in a way that doesn't represent an API compatibility break. I need to think a bit harder about how that might be achieved. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
743384829 | |
https://github.com/simonw/sqlite-utils/issues/217#issuecomment-753566184 | https://api.github.com/repos/simonw/sqlite-utils/issues/217 | 753566184 | MDEyOklzc3VlQ29tbWVudDc1MzU2NjE4NA== | 9599 | 2021-01-03T04:27:38Z | 2021-01-03T04:27:38Z | OWNER | Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#quoting-strings-for-use-in-sql | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777543336 | |
https://github.com/simonw/sqlite-utils/issues/216#issuecomment-753566156 | https://api.github.com/repos/simonw/sqlite-utils/issues/216 | 753566156 | MDEyOklzc3VlQ29tbWVudDc1MzU2NjE1Ng== | 9599 | 2021-01-03T04:27:14Z | 2021-01-03T04:27:14Z | OWNER | Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#introspection | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777540352 | |
https://github.com/simonw/sqlite-utils/issues/218#issuecomment-753563757 | https://api.github.com/repos/simonw/sqlite-utils/issues/218 | 753563757 | MDEyOklzc3VlQ29tbWVudDc1MzU2Mzc1Nw== | 9599 | 2021-01-03T03:49:51Z | 2021-01-03T03:49:51Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#listing-triggers | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777560474 | |
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545757 | https://api.github.com/repos/simonw/sqlite-utils/issues/215 | 753545757 | MDEyOklzc3VlQ29tbWVudDc1MzU0NTc1Nw== | 9599 | 2021-01-02T23:58:07Z | 2021-01-02T23:58:07Z | OWNER | Thought: maybe there should be a `.reset_counts()` method too, for if the table gets out of date with the triggers. One way that could happen is if a table is dropped and recreated - the counts in the `_counts` table would likely no longer match the number of rows in that table. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777535402 | |
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545381 | https://api.github.com/repos/simonw/sqlite-utils/issues/215 | 753545381 | MDEyOklzc3VlQ29tbWVudDc1MzU0NTM4MQ== | 9599 | 2021-01-02T23:52:52Z | 2021-01-02T23:52:52Z | OWNER | Idea: a `db.cached_counts()` method that returns a dictionary of data from the `_counts` table. Call it with a list of tables to get back the counts for just those tables. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777535402 | |
https://github.com/simonw/sqlite-utils/issues/217#issuecomment-753544914 | https://api.github.com/repos/simonw/sqlite-utils/issues/217 | 753544914 | MDEyOklzc3VlQ29tbWVudDc1MzU0NDkxNA== | 9599 | 2021-01-02T23:47:42Z | 2021-01-02T23:47:42Z | OWNER | https://github.com/simonw/sqlite-utils/blob/9a5c92b63e7917c93cc502478493c51c781b2ecc/sqlite_utils/db.py#L231-L239 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777543336 | |
https://github.com/simonw/sqlite-utils/issues/213#issuecomment-753535488 | https://api.github.com/repos/simonw/sqlite-utils/issues/213 | 753535488 | MDEyOklzc3VlQ29tbWVudDc1MzUzNTQ4OA== | 9599 | 2021-01-02T22:03:48Z | 2021-01-02T22:03:48Z | OWNER | I got this error while prototyping this: too many levels of trigger recursion It looks like that's because SQLite doesn't like triggers on a table that themselves then update that table - so I'm going to exclude the `_counts` table from this mechanism. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777529979 | |
https://github.com/simonw/sqlite-utils/issues/213#issuecomment-753533775 | https://api.github.com/repos/simonw/sqlite-utils/issues/213 | 753533775 | MDEyOklzc3VlQ29tbWVudDc1MzUzMzc3NQ== | 9599 | 2021-01-02T21:47:10Z | 2021-01-02T21:47:10Z | OWNER | I'm going to skip virtual tables, which I can identify using this property: https://github.com/simonw/sqlite-utils/blob/1cad7fad3e7a5b734088f5cc545b69a055e636da/sqlite_utils/db.py#L720-L726 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777529979 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753524779 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753524779 | MDEyOklzc3VlQ29tbWVudDc1MzUyNDc3OQ== | 9599 | 2021-01-02T20:19:26Z | 2021-01-02T20:19:26Z | OWNER | Idea: version the metadata scheme. If the table is called `_metadata_v1` it gives me a clear path to designing a new scheme in the future. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 | |
https://github.com/simonw/sqlite-utils/issues/212#issuecomment-753422324 | https://api.github.com/repos/simonw/sqlite-utils/issues/212 | 753422324 | MDEyOklzc3VlQ29tbWVudDc1MzQyMjMyNA== | 9599 | 2021-01-02T03:00:34Z | 2021-01-02T03:00:34Z | OWNER | Here's a prototype: ```python with db.conn: db.conn.executescript(""" CREATE TABLE IF NOT EXISTS [_counts] ([table] TEXT PRIMARY KEY, [count] INTEGER DEFAULT 0); CREATE TRIGGER IF NOT EXISTS [Street_Tree_List_counts_ai] AFTER INSERT ON [Street_Tree_List] BEGIN INSERT OR REPLACE INTO _counts VALUES ('Street_Tree_List', COALESCE( (SELECT count FROM _counts WHERE [table]='Street_Tree_List'), 0) + 1); END; CREATE TRIGGER IF NOT EXISTS [Street_Tree_List_counts_ad] AFTER DELETE ON [Street_Tree_List] BEGIN INSERT OR REPLACE INTO _counts VALUES ('Street_Tree_List', COALESCE( (SELECT count FROM _counts WHERE [table]='Street_Tree_List'), 0) - 1); END; INSERT OR REPLACE INTO _counts VALUES ('Street_Tree_List', (select count(*) from [Street_Tree_List])); """) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777392020 | |
https://github.com/simonw/sqlite-utils/issues/210#issuecomment-753406744 | https://api.github.com/repos/simonw/sqlite-utils/issues/210 | 753406744 | MDEyOklzc3VlQ29tbWVudDc1MzQwNjc0NA== | 9599 | 2021-01-02T00:02:39Z | 2021-01-02T00:02:39Z | OWNER | It looks like https://github.com/ofajardo/pyreadr is a good library for this. I won't add this to `sqlite-utils` because it's quite a bulky dependency for a relatively small feature. Normally I'd write a `rdata-to-sqlite` tool similar to https://pypi.org/project/dbf-to-sqlite/ - but I'm actually working on a new plugin hook for Datasette that might be an even better fit for this. The idea is to allow Datasette plugins to define input formats - such as RData - which would then result in being able to import them on the command-line with `datasette insert my.db file.rdata` or by uploading a file through the Datasette web interface. That work is happening over here: https://github.com/simonw/datasette/issues/1160 - I'll close this issue in favour of a sometime-in-the-future `datasette-import-rdata` plugin. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
767685961 | |
https://github.com/simonw/sqlite-utils/issues/209#issuecomment-753405835 | https://api.github.com/repos/simonw/sqlite-utils/issues/209 | 753405835 | MDEyOklzc3VlQ29tbWVudDc1MzQwNTgzNQ== | 9599 | 2021-01-01T23:52:06Z | 2021-01-01T23:52:06Z | OWNER | I just hit this one too. Such a weird bug! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
766156875 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753402423 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753402423 | MDEyOklzc3VlQ29tbWVudDc1MzQwMjQyMw== | 9599 | 2021-01-01T23:16:05Z | 2021-01-01T23:16:05Z | OWNER | One catch: solving the "show me all metadata for everything in this Datasette instance" problem. Ideally there would be a SQLite table that can be queried for this. But the need to resolve the potentially complex set of precedence rules means that table would be difficult if not impossible to provide at run-time. Ideally a denormalized table would be available that featured the results of running those precedence rule calculations. But how to handle keeping this up-to-date? It would need to be recalculated any time a `_metadata` table in any of the attached databases had an update. This is a much larger problem - but one potential fix would be to use triggers to maintain a "version number" for the `_metadata` table - similar to SQLite's own built-in `schema_version` mechanism. Triggers could increment a counter any time a record in that table was added, deleted or updated. Such a mechanism would have applications outside of just this `_metadata` system. The ability to attach a version number to any table and have it automatically incremented when that table changes (via triggers) could help with all kinds of other Datasette-at-scale problems, including things like cached table counts. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753401001 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753401001 | MDEyOklzc3VlQ29tbWVudDc1MzQwMTAwMQ== | 9599 | 2021-01-01T23:01:45Z | 2021-01-01T23:01:45Z | OWNER | I need to prototype this. Could I do that as a plugin? I think so - I could try out the algorithm for loading metadata and display it on pages using some custom templates. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753400420 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753400420 | MDEyOklzc3VlQ29tbWVudDc1MzQwMDQyMA== | 9599 | 2021-01-01T22:53:58Z | 2021-01-01T22:53:58Z | OWNER | Precedence idea: - First priority is non-_internal metadata from other databases - if those conflict then pick then the alphabetically-ordered-first database name wins - Next priority: `_internal` metadata, which should have been loaded from `metadata.json` - Last priority: the `_metadata` table from that database itself, i.e. the default "baked in" metadata | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753400306 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753400306 | MDEyOklzc3VlQ29tbWVudDc1MzQwMDMwNg== | 9599 | 2021-01-01T22:52:44Z | 2021-01-01T22:52:44Z | OWNER | Also: probably load column metadata as part of the table metadata rather than loading column metadata individually, since it's going to be rare to want the metadata for a single column rather than for an entire table full of columns. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753400265 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753400265 | MDEyOklzc3VlQ29tbWVudDc1MzQwMDI2NQ== | 9599 | 2021-01-01T22:52:09Z | 2021-01-01T22:52:09Z | OWNER | From an implementation perspective, I think the way this works is SQL queries read the relevant metadata from ALL available metadata tables, then Python code solves the precedence rules to produce the final, combined metadata for a database/table/column. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753399635 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753399635 | MDEyOklzc3VlQ29tbWVudDc1MzM5OTYzNQ== | 9599 | 2021-01-01T22:45:21Z | 2021-01-01T22:50:21Z | OWNER | Would also need to figure out the precedence rules: - What happens if the database has a `_metadata` table with data that conflicts with a remote metadata record from another database? I think the other database should win, because that allows plugins to over-ride the default metadata for something. - Do JSON values get merged together? So if one table provides a description and another provides a title do both values get returned? - If a database has a `license`, does that "cascade" down to the tables? What about `source` and `about`? - What if there are two databases (or more) that provide conflicting metadata for a table in some other database? Also, `_internal` may have loaded data from `metadata.json` that conflicts with some other remote table metadata definition. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753399428 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753399428 | MDEyOklzc3VlQ29tbWVudDc1MzM5OTQyOA== | 9599 | 2021-01-01T22:43:14Z | 2021-01-01T22:43:22Z | OWNER | Could this use a compound primary key on `database, table, column`? Does that work with null values? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753399366 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753399366 | MDEyOklzc3VlQ29tbWVudDc1MzM5OTM2Ng== | 9599 | 2021-01-01T22:42:37Z | 2021-01-01T22:42:37Z | OWNER | So what would the database schema for this look like? I'm leaning towards a single table called `_metadata`, because that's a neater fit for baking the metadata into the database file along with the data that it is describing. Alternatively I could have multiple tables sharing that prefix - `_metadata_database` and `_metadata_tables` and `_metadata_columns` perhaps. If it's just a single `_metadata` table, the schema could look like this: | database | table | column | metadata | | --- | --- | --- | --- | | | mytable | | {"title": "My Table" } | | | mytable | mycolumn | {"description": "Column description" } | | otherdb | othertable | | {"description": "Table in another DB" } | If the `database` column is `null` it means "this is describing a table in the same database file as this `_metadata` table". The alternative to the `metadata` JSON column would be separate columns for each potential metadata value - `license`, `source`, `about`, `about_url` etc. But that makes it harder for people to create custom metadata fields. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753398542 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753398542 | MDEyOklzc3VlQ29tbWVudDc1MzM5ODU0Mg== | 9599 | 2021-01-01T22:37:24Z | 2021-01-01T22:37:24Z | OWNER | The direction I'm leaning in now is the following: - Metadata always lives in SQLite tables - These tables can be co-located with the database they describe (same DB file) - ... or they can be in a different DB file and reference the other database that they are describing - Metadata provided on startup in a `metadata.json` file is loaded into an in-memory metadata table using that same mechanism Plugins that want to provide metadata can do so by populating a table. They could even maintain their own in-memory database for this, or they could write to the `_internal` in-memory database, or they could write to a table in a database on disk. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 | |
https://github.com/simonw/datasette/issues/1168#issuecomment-753392102 | https://api.github.com/repos/simonw/datasette/issues/1168 | 753392102 | MDEyOklzc3VlQ29tbWVudDc1MzM5MjEwMg== | 9599 | 2021-01-01T22:06:33Z | 2021-01-01T22:06:33Z | OWNER | Some SQLite databases include SQL comments in the schema definition which tell you what each column means: ```sql CREATE TABLE User -- A table comment ( uid INTEGER, -- A field comment flags INTEGER -- Another field comment ); ``` The problem with these is that they're not exposed to SQLite in any mechanism other than parsing the `CREATE TABLE` statement from the `sqlite_master` table to extract those columns. I had an idea to build a plugin that could return these. That would be easy with a "get metadata for this column" plugin hook - in the absence of one a plugin could still run that reads the schemas on startup and uses them to populate a metadata database table somewhere. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
777333388 |