html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/issues/943#issuecomment-696777886,https://api.github.com/repos/simonw/datasette/issues/943,696777886,MDEyOklzc3VlQ29tbWVudDY5Njc3Nzg4Ng==,9599,2020-09-22T14:58:54Z,2020-09-22T14:58:54Z,OWNER,"```python class DatasetteClient: def __init__(self, ds): self._client = httpx.AsyncClient(app=ds.app()) def _fix(self, path): if path.startswith(""/""): path = ""http://localhost{}"".format(path) return path async def get(self, path, **kwargs): return await self._client.get(self._fix(path), **kwargs) async def options(self, path, **kwargs): return await self._client.options(self._fix(path), **kwargs) async def head(self, path, **kwargs): return await self._client.head(self._fix(path), **kwargs) async def post(self, path, **kwargs): return await self._client.post(self._fix(path), **kwargs) async def put(self, path, **kwargs): return await self._client.put(self._fix(path), **kwargs) async def patch(self, path, **kwargs): return await self._client.patch(self._fix(path), **kwargs) async def delete(self, path, **kwargs): return await self._client.delete(self._fix(path), **kwargs) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-696776828,https://api.github.com/repos/simonw/datasette/issues/943,696776828,MDEyOklzc3VlQ29tbWVudDY5Njc3NjgyOA==,9599,2020-09-22T14:57:13Z,2020-09-22T14:57:13Z,OWNER,"I may as well implement all of the HTTP methods supported by the `httpx` client: - get - options - head - post - put - patch - delete","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-696775516,https://api.github.com/repos/simonw/datasette/issues/943,696775516,MDEyOklzc3VlQ29tbWVudDY5Njc3NTUxNg==,9599,2020-09-22T14:55:10Z,2020-09-22T14:55:10Z,OWNER,"Even smaller `DatasetteClient` implementation: ```python class DatasetteClient: def __init__(self, ds): self._client = httpx.AsyncClient(app=ds.app()) def _fix(self, path): if path.startswith(""/""): path = ""http://localhost{}"".format(path) return path async def get(self, path, **kwargs): return await self._client.get(self._fix(path), **kwargs) async def post(self, path, **kwargs): return await self._client.post(self._fix(path), **kwargs) async def options(self, path, **kwargs): return await self._client.options(self._fix(path), **kwargs) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-696774711,https://api.github.com/repos/simonw/datasette/issues/943,696774711,MDEyOklzc3VlQ29tbWVudDY5Njc3NDcxMQ==,9599,2020-09-22T14:53:56Z,2020-09-22T14:53:56Z,OWNER,"How important is it to use `httpx.AsyncClient` with a context manager? https://www.python-httpx.org/async/#opening-and-closing-clients says: > Alternatively, use `await client.aclose()` if you want to close a client explicitly: > > ``` > client = httpx.AsyncClient() > ... > await client.aclose() > ``` The `.aclose()` method has a comment saying ""Close transport and proxies"" - I'm not using proxies, so the relevant implementation seems to be a call to `await self._transport.aclose()` in https://github.com/encode/httpx/blob/f932af9172d15a803ad40061a4c2c0cd891645cf/httpx/_client.py#L1741-L1751 The transport I am using is a class called `ASGITransport` in https://github.com/encode/httpx/blob/master/httpx/_transports/asgi.py The `aclose()` method on that class does nothing. So it looks like I can instantiate a client without bothering with the `async with httpx.AsyncClient` bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-696769853,https://api.github.com/repos/simonw/datasette/issues/943,696769853,MDEyOklzc3VlQ29tbWVudDY5Njc2OTg1Mw==,9599,2020-09-22T14:46:21Z,2020-09-22T14:46:21Z,OWNER,This adds `httpx` as a dependency - I think I'm OK with that. I use it for testing in all of my plugins anyway.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-696769501,https://api.github.com/repos/simonw/datasette/issues/943,696769501,MDEyOklzc3VlQ29tbWVudDY5Njc2OTUwMQ==,9599,2020-09-22T14:45:49Z,2020-09-22T14:45:49Z,OWNER,"I put together a minimal prototype of this and it feels pretty good: ```diff diff --git a/datasette/app.py b/datasette/app.py index 20aae7d..fb3bdad 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -4,6 +4,7 @@ import collections import datetime import glob import hashlib +import httpx import inspect import itertools from itsdangerous import BadSignature @@ -312,6 +313,7 @@ class Datasette: self._register_renderers() self._permission_checks = collections.deque(maxlen=200) self._root_token = secrets.token_hex(32) + self.client = DatasetteClient(self) async def invoke_startup(self): for hook in pm.hook.startup(datasette=self): @@ -1209,3 +1211,25 @@ def route_pattern_from_filepath(filepath): class NotFoundExplicit(NotFound): pass + + +class DatasetteClient: + def __init__(self, ds): + self.app = ds.app() + + def _fix(self, path): + if path.startswith(""/""): + path = ""http://localhost{}"".format(path) + return path + + async def get(self, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.get(self._fix(path), **kwargs) + + async def post(self, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.post(self._fix(path), **kwargs) + + async def options(self, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.options(self._fix(path), **kwargs) ``` Used like this in `ipython`: ``` In [1]: from datasette.app import Datasette In [2]: ds = Datasette([""fixtures.db""]) In [3]: (await ds.client.get(""/-/config.json"")).json() Out[3]: {'default_page_size': 100, 'max_returned_rows': 1000, 'num_sql_threads': 3, 'sql_time_limit_ms': 1000, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'hash_urls': False, 'allow_facet': True, 'allow_download': True, 'suggest_facets': True, 'default_cache_ttl': 5, 'default_cache_ttl_hashed': 31536000, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'base_url': '/'} In [4]: (await ds.client.get(""/fixtures/facetable.json?_shape=array"")).json() Out[4]: [{'pk': 1, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', 'city_id': 1, 'neighborhood': 'Mission', 'tags': '[""tag1"", ""tag2""]', 'complex_array': '[{""foo"": ""bar""}]', 'distinct_some_null': 'one'}, {'pk': 2, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', 'city_id': 1, 'neighborhood': 'Dogpatch', 'tags': '[""tag1"", ""tag3""]', 'complex_array': '[]', 'distinct_some_null': 'two'}, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-693009048,https://api.github.com/repos/simonw/datasette/issues/943,693009048,MDEyOklzc3VlQ29tbWVudDY5MzAwOTA0OA==,9599,2020-09-15T22:17:30Z,2020-09-22T14:37:00Z,OWNER,"Maybe instead of implementing `datasette.get()` and `datasette.post()` and `datasette.request()` and `datasette.stream()` I could instead have a nested object called `datasette.client` which is a preconfigured `AsyncClient` instance. ```python response = await datasette.client.get(""/"") ``` Or perhaps this should be a method in case I ever need to be able to `await` it: ```python response = await (await datasette.client()).get(""/"") ``` This is a bit cosmetically ugly though, I'd rather avoid that if possible. Maybe I could get this working by returning an object from `.client()` which provides a `await obj.get()` method: ```python response = await datasette.client().get(""/"") ``` I don't think there's any benefit to that over `await datasette.client.get()` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/sqlite-utils/issues/168#issuecomment-696573944,https://api.github.com/repos/simonw/sqlite-utils/issues/168,696573944,MDEyOklzc3VlQ29tbWVudDY5NjU3Mzk0NA==,9599,2020-09-22T08:11:30Z,2020-09-22T08:11:30Z,OWNER,Huh... maybe I don't need to do anything here? It looks like it's been kept up to date: https://github.com/Homebrew/homebrew-core/commits/master/Formula/sqlite-utils.rb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706167456, https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696567988,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696567988,MDEyOklzc3VlQ29tbWVudDY5NjU2Nzk4OA==,9599,2020-09-22T07:57:50Z,2020-09-22T07:57:50Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#transforming-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416, https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696567460,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696567460,MDEyOklzc3VlQ29tbWVudDY5NjU2NzQ2MA==,9599,2020-09-22T07:56:42Z,2020-09-22T07:56:42Z,OWNER,`.transform()` has landed now which should make this a lot easier to solve.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929, https://github.com/simonw/sqlite-utils/issues/26#issuecomment-696566750,https://api.github.com/repos/simonw/sqlite-utils/issues/26,696566750,MDEyOklzc3VlQ29tbWVudDY5NjU2Njc1MA==,9599,2020-09-22T07:55:00Z,2020-09-22T07:55:00Z,OWNER,"Problem: `extract` means something else now, see #47 and the upcoming work in #42.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455486286, https://github.com/simonw/sqlite-utils/issues/167#issuecomment-696565981,https://api.github.com/repos/simonw/sqlite-utils/issues/167,696565981,MDEyOklzc3VlQ29tbWVudDY5NjU2NTk4MQ==,9599,2020-09-22T07:53:13Z,2020-09-22T07:53:13Z,OWNER,"Confirmed this is a bug, https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes explicitly says you should do the `PRAGMA foreign_keys` bits before and after the transaction, not during. Right now my code does this INSIDE the transaction: https://github.com/simonw/sqlite-utils/blob/f29f6821f2d08e91c5c6d65d885a1bbc0c743bdd/sqlite_utils/db.py#L790-L793 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706098005, https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696520928,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696520928,MDEyOklzc3VlQ29tbWVudDY5NjUyMDkyOA==,9599,2020-09-22T05:50:17Z,2020-09-22T05:50:17Z,OWNER,"Idea for CLI options: ``` --type age integer --drop colname --rename oldname newname --not-null col --not-null-false col --pk new_id --pk-none --default col value --default-none column --drop-foreign-key col other_table other_column ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416, https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696500922,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696500922,MDEyOklzc3VlQ29tbWVudDY5NjUwMDkyMg==,9599,2020-09-22T04:22:40Z,2020-09-22T04:22:40Z,OWNER,Documentation for the `.transform()` method #114 (now landed) is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696500767,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696500767,MDEyOklzc3VlQ29tbWVudDY5NjUwMDc2Nw==,9599,2020-09-22T04:21:45Z,2020-09-22T04:21:45Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696494070,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696494070,MDEyOklzc3VlQ29tbWVudDY5NjQ5NDA3MA==,9599,2020-09-22T03:48:58Z,2020-09-22T03:48:58Z,OWNER,"One last thing. https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_change says that the first step should be: > If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. And the last steps should be: > If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints. > > Commit the transaction started in step 2. > > If foreign keys constraints were originally enabled, reenable them now. I need to implement that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696490851,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696490851,MDEyOklzc3VlQ29tbWVudDY5NjQ5MDg1MQ==,9599,2020-09-22T03:33:54Z,2020-09-22T03:33:54Z,OWNER,It would be neat if `.transform(pk=None)` converted a primary key table to a rowid table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696488201,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696488201,MDEyOklzc3VlQ29tbWVudDY5NjQ4ODIwMQ==,9599,2020-09-22T03:21:16Z,2020-09-22T03:21:16Z,OWNER,Just needs documentation now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696485791,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696485791,MDEyOklzc3VlQ29tbWVudDY5NjQ4NTc5MQ==,9599,2020-09-22T03:10:15Z,2020-09-22T03:10:15Z,OWNER,"Design decision needed on foreign keys: what does the syntax look like for removing an existing foreign key? Since I already have a good implementation of `add_foreign_key()` I'm tempted to only support dropping them. Maybe like this: ```python table.transform(drop_foreign_keys=[(""author_id"", ""author"", ""id"")]) ``` It's a bit crufty but it's such a rare use-case that I think this will be good enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696480925,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696480925,MDEyOklzc3VlQ29tbWVudDY5NjQ4MDkyNQ==,9599,2020-09-22T02:45:47Z,2020-09-22T02:45:47Z,OWNER,"I'm not going to do `conversions=` because it would be inconsistent with how they work elsewhere. The SQL generated by this function looks like this: INSERT INTO dogs_new_tmp VALUES (a, b) SELECT a, b from dogs; So passing `conversions={""name"": ""upper(?)""})` wouldn't make sense, since we're not using arguments hence there is no-where for that `?` to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696473559,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696473559,MDEyOklzc3VlQ29tbWVudDY5NjQ3MzU1OQ==,9599,2020-09-22T02:10:37Z,2020-09-22T02:10:37Z,OWNER,"Maybe something like this: sqlite-utils transform mydb.db mytable -c age integer --rename age dog_age ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416, https://github.com/simonw/sqlite-utils/issues/163#issuecomment-696465788,https://api.github.com/repos/simonw/sqlite-utils/issues/163,696465788,MDEyOklzc3VlQ29tbWVudDY5NjQ2NTc4OA==,9599,2020-09-22T01:33:04Z,2020-09-22T01:33:04Z,OWNER,This would apply to `.transform()` in #114 too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696454485,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696454485,MDEyOklzc3VlQ29tbWVudDY5NjQ1NDQ4NQ==,9599,2020-09-22T00:42:35Z,2020-09-22T00:42:35Z,OWNER,The reason I'm working on this now is that I'd like to support many more options for data cleanup in the Datasette ecosystem - so being able to do things like convert the type of existing columns becomes increasingly important.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696454084,https://api.github.com/repos/simonw/sqlite-utils/issues/162,696454084,MDEyOklzc3VlQ29tbWVudDY5NjQ1NDA4NA==,9599,2020-09-22T00:40:44Z,2020-09-22T00:40:44Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#registering-custom-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705995722, https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696449345,https://api.github.com/repos/simonw/sqlite-utils/issues/162,696449345,MDEyOklzc3VlQ29tbWVudDY5NjQ0OTM0NQ==,9599,2020-09-22T00:22:46Z,2020-09-22T00:22:46Z,OWNER,Inspired by the idea of adding `conversions=` to #114 - since this would make it easy to register custom Python functions that can be used to convert the values in a table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705995722, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696446658,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696446658,MDEyOklzc3VlQ29tbWVudDY5NjQ0NjY1OA==,9599,2020-09-22T00:13:55Z,2020-09-22T00:14:21Z,OWNER,"Idea: allow a `conversions=` parameter, as seen on `.insert_all()` and friends, which lets you apply a SQL transformation function as part of the operation. E.g.: ```python table.transform({""age"": int}, conversions={""name"": ""upper(?)""}) ``` https://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696445766,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696445766,MDEyOklzc3VlQ29tbWVudDY5NjQ0NTc2Ng==,9599,2020-09-22T00:10:50Z,2020-09-22T00:11:12Z,OWNER,"A less horrible interface might be the following: ```python # Ensure the 'age' column is not null: table.transform(not_null={""age""}) # The 'age' column is not null but I don't want it to be: table.transform(not_null={""age"": False}) ``` So if the argument is a set it means ""make sure these are all not null"" - if the argument is a dictionary it means ""set these to be null or not null depending on if their dictionary value is true or false"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444842,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696444842,MDEyOklzc3VlQ29tbWVudDY5NjQ0NDg0Mg==,9599,2020-09-22T00:07:43Z,2020-09-22T00:09:05Z,OWNER,"Syntax challenge: I could use `.transform(defaults={""age"": None})` to indicate that the `age` column should have its default removed, but how would I tell `.transform()` that the `age` column, currently `not null`, should have the `not null` removed from it? I could do this: `.transform(not_not_null={""age""})` - it's a bit gross but it's also kind of funny. I actually like it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444353,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696444353,MDEyOklzc3VlQ29tbWVudDY5NjQ0NDM1Mw==,9599,2020-09-22T00:06:12Z,2020-09-22T00:06:12Z,OWNER,I should support `not_null=` and `default=` arguments to the `.transform()` method because it looks like you can't use `ALTER TABLE` to change those.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443845,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443845,MDEyOklzc3VlQ29tbWVudDY5NjQ0Mzg0NQ==,9599,2020-09-22T00:04:31Z,2020-09-22T00:04:44Z,OWNER,"Good news: the `.columns` introspection does tell me those things: ``` >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db.create_table(""foo"", {""id"": int, ""name"": str, ""age"": int}, defaults={""age"": 1}, not_null={""name"", ""age""}) >>> db[""foo""]
>>> print(db[""foo""].schema) CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT NOT NULL, [age] INTEGER NOT NULL DEFAULT 1 ) >>> db[""foo""].columns [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=None, is_pk=0), Column(cid=2, name='age', type='INTEGER', notnull=1, default_value='1', is_pk=0)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443190,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443190,MDEyOklzc3VlQ29tbWVudDY5NjQ0MzE5MA==,9599,2020-09-22T00:02:22Z,2020-09-22T00:02:22Z,OWNER,How would I detect which columns are `not_null` and what their defaults are? I don`t think my introspection logic handles that yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443042,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443042,MDEyOklzc3VlQ29tbWVudDY5NjQ0MzA0Mg==,9599,2020-09-22T00:01:50Z,2020-09-22T00:01:50Z,OWNER,"When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about `hash_id` or `extracts=` since those don't affect the structure of the table as it is being created - well, `hash_id` does but if we are transforming an existing table we will get the `hash_id` column for free.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696442621,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696442621,MDEyOklzc3VlQ29tbWVudDY5NjQ0MjYyMQ==,9599,2020-09-22T00:00:23Z,2020-09-22T00:00:23Z,OWNER,I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696435194,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696435194,MDEyOklzc3VlQ29tbWVudDY5NjQzNTE5NA==,9599,2020-09-21T23:34:14Z,2020-09-21T23:35:00Z,OWNER,"I think the fiddliest part of the implementation here is code that takes the existing `columns_dict` of the table and the incoming `columns=` and `drop=` and `rename=` parameters and produces the columns dictionary for the new table, ready to be fed to `.create_table()`. This logic probably also needs to return a structure that can be used to build the `INSERT INTO ... SELECT ... FROM` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434638,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434638,MDEyOklzc3VlQ29tbWVudDY5NjQzNDYzOA==,9599,2020-09-21T23:32:26Z,2020-09-21T23:32:26Z,OWNER,A test that confirms that this mechanism can turn a `rowid` into a non-rowid table would be good too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434237,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434237,MDEyOklzc3VlQ29tbWVudDY5NjQzNDIzNw==,9599,2020-09-21T23:31:07Z,2020-09-21T23:31:57Z,OWNER,"Does it make sense to support the `pk=` argument for changing the primary key? If the user requests a primary key that doesn't make sense I think an integrity error will be raised when the SQL is being executed, which should hopefully cancel the transaction and raise an error. Need to check that this is what happens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434097,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434097,MDEyOklzc3VlQ29tbWVudDY5NjQzNDA5Nw==,9599,2020-09-21T23:30:40Z,2020-09-21T23:30:40Z,OWNER,"Since I have a `column_order=None` argument already, maybe I can ignore the order of the columns in that first argument and use that instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433778,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696433778,MDEyOklzc3VlQ29tbWVudDY5NjQzMzc3OA==,9599,2020-09-21T23:29:39Z,2020-09-21T23:29:39Z,OWNER,"The `columns=` argument is optional - so you can do just a rename operation like so: ``` table.transform(rename={""age"": ""dog_age""}) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433542,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696433542,MDEyOklzc3VlQ29tbWVudDY5NjQzMzU0Mg==,9599,2020-09-21T23:28:58Z,2020-09-21T23:28:58Z,OWNER,"If you want to both change the type of a column AND rename it in the same operation, how would you do that? I think like this: ```python table.transform({""age"": int}, rename={""age"": ""dog_age""}) ``` So any rename logic is applied at the end, after the type transformation or re-ordering logic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696432690,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696432690,MDEyOklzc3VlQ29tbWVudDY5NjQzMjY5MA==,9599,2020-09-21T23:26:32Z,2020-09-21T23:27:38Z,OWNER,"To expand on what that first argument - the `columns` argument - does. Say you have a table like this: ``` id integer name text age text ``` Any columns omitted from the `columns=` argument are left alone - they have to be explicitly dropped using `drop=` if you want to drop them. Any new columns are added (at the end of the table): ``` table.tranform({""size"": float}) ``` Any columns that have their type changed will have their type changed: ``` table.tranform({""age"": int}) ``` Should I also re-order columns if the order doesn't match? I think so. Open question as to what happens to columns that aren't mentioned at all in the dictionary though - what order should they go in?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696431058,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696431058,MDEyOklzc3VlQ29tbWVudDY5NjQzMTA1OA==,9599,2020-09-21T23:21:37Z,2020-09-21T23:21:37Z,OWNER,I may need to do something special for `rowid` tables to ensure that the `rowid` values in the transformed table match those from the old table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696430843,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696430843,MDEyOklzc3VlQ29tbWVudDY5NjQzMDg0Mw==,9599,2020-09-21T23:21:00Z,2020-09-21T23:21:00Z,OWNER,"For FTS tables associated with the table that is being transformed, should I automatically drop the old FTS table and recreate it against the new one or will it just magically continue to work after the table is renamed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423138,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696423138,MDEyOklzc3VlQ29tbWVudDY5NjQyMzEzOA==,9599,2020-09-21T22:59:17Z,2020-09-21T23:01:06Z,OWNER,I'm going to sketch out a prototype of this new API design in that branch.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423066,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696423066,MDEyOklzc3VlQ29tbWVudDY5NjQyMzA2Ng==,9599,2020-09-21T22:59:01Z,2020-09-21T22:59:01Z,OWNER,"I'm rethinking the API design now. Maybe it could look like this: To change the type of the `author_id` column from `text` to `int`: ```python books.transform({""author_id"": int}) ``` This would leave the existing columns alone, but would change the type of this column. To rename `author_id` to `author_identifier`: ```python books.transform(rename={""author_id"": ""author_identifier""}) ``` To drop a column: ```python books.transform(drop=[""author_id""]) ``` Since the parameters all operate on columns they don't need to be called `drop_column` and `rename_column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696421240,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696421240,MDEyOklzc3VlQ29tbWVudDY5NjQyMTI0MA==,9599,2020-09-21T22:53:48Z,2020-09-21T22:53:48Z,OWNER,"I've decided to call this `table.transform()` - I was over-thinking whether people would remember that `.transform()` actually transforms the table, but that's what documentation is for.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740, https://github.com/simonw/datasette/issues/972#issuecomment-696308847,https://api.github.com/repos/simonw/datasette/issues/972,696308847,MDEyOklzc3VlQ29tbWVudDY5NjMwODg0Nw==,9599,2020-09-21T19:01:25Z,2020-09-21T19:01:25Z,OWNER,I did a bunch of initial work for this in #427.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705840673, https://github.com/simonw/datasette/issues/971#issuecomment-696307922,https://api.github.com/repos/simonw/datasette/issues/971,696307922,MDEyOklzc3VlQ29tbWVudDY5NjMwNzkyMg==,9599,2020-09-21T18:59:52Z,2020-09-21T19:00:02Z,OWNER,"Given `dbstat` isn't as widely available as I thought I'm going to let people who want to use `dbstat` run their own `select * from dbstat` queries rather than bake support directly into Datasette. The experience of exploring `dbstat` will improve if I land support for running facets against arbitrary custom SQL queries, which is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646 https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457, https://github.com/simonw/datasette/issues/971#issuecomment-696304108,https://api.github.com/repos/simonw/datasette/issues/971,696304108,MDEyOklzc3VlQ29tbWVudDY5NjMwNDEwOA==,9599,2020-09-21T18:52:50Z,2020-09-21T18:52:50Z,OWNER,Looks like the `pysqlite3-binary` package doesn't support `dbstat` either.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457, https://github.com/simonw/datasette/issues/971#issuecomment-696302868,https://api.github.com/repos/simonw/datasette/issues/971,696302868,MDEyOklzc3VlQ29tbWVudDY5NjMwMjg2OA==,9599,2020-09-21T18:50:40Z,2020-09-21T18:50:40Z,OWNER,Easiest way to get this may be to run `create view dbstat_view as select * from dbstat` on databases that support it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457, https://github.com/simonw/datasette/issues/971#issuecomment-696302020,https://api.github.com/repos/simonw/datasette/issues/971,696302020,MDEyOklzc3VlQ29tbWVudDY5NjMwMjAyMA==,9599,2020-09-21T18:49:09Z,2020-09-21T18:49:09Z,OWNER,... made harder to work on because I apparently don't have the `DBSTAT_VTAB` module on macOS.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457, https://github.com/simonw/datasette/issues/971#issuecomment-696298614,https://api.github.com/repos/simonw/datasette/issues/971,696298614,MDEyOklzc3VlQ29tbWVudDY5NjI5ODYxNA==,9599,2020-09-21T18:43:07Z,2020-09-21T18:43:07Z,OWNER,"Or, do this: ```sql SELECT 1 FROM dbstat limit 1; ``` And see if it returns a ""table does not exist"" error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457, https://github.com/simonw/datasette/issues/971#issuecomment-696297930,https://api.github.com/repos/simonw/datasette/issues/971,696297930,MDEyOklzc3VlQ29tbWVudDY5NjI5NzkzMA==,9599,2020-09-21T18:41:47Z,2020-09-21T18:41:47Z,OWNER,"https://www.sqlite.org/dbstat.html > The DBSTAT virtual table is an eponymous virtual table, meaning that is not necessary to run CREATE VIRTUAL TABLE to create an instance of the dbstat virtual table before using it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457, https://github.com/simonw/datasette/issues/971#issuecomment-696297601,https://api.github.com/repos/simonw/datasette/issues/971,696297601,MDEyOklzc3VlQ29tbWVudDY5NjI5NzYwMQ==,9599,2020-09-21T18:41:07Z,2020-09-21T18:41:07Z,OWNER,"How to detect it? Looks like it's visible in SQLite compile time options: https://latest.datasette.io/-/versions ``` ""compile_options"": [ ""COMPILER=gcc-8.3.0"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_DBSTAT_VTAB"", ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457, https://github.com/simonw/datasette/issues/670#issuecomment-696163452,https://api.github.com/repos/simonw/datasette/issues/670,696163452,MDEyOklzc3VlQ29tbWVudDY5NjE2MzQ1Mg==,652285,2020-09-21T14:46:10Z,2020-09-21T14:46:10Z,NONE,I'm currently using PostgREST to serve OpenAPI APIs off Postgresql databases. I would like to try out datasette once this becomes available on Postgres.,"{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564833696, https://github.com/simonw/datasette/issues/970#issuecomment-695896557,https://api.github.com/repos/simonw/datasette/issues/970,695896557,MDEyOklzc3VlQ29tbWVudDY5NTg5NjU1Nw==,9599,2020-09-21T04:40:12Z,2020-09-21T04:40:12Z,OWNER,The Python standard library has a module for this: https://docs.python.org/3/library/webbrowser.html,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492, https://github.com/simonw/datasette/issues/970#issuecomment-695895960,https://api.github.com/repos/simonw/datasette/issues/970,695895960,MDEyOklzc3VlQ29tbWVudDY5NTg5NTk2MA==,9599,2020-09-21T04:36:45Z,2020-09-21T04:36:45Z,OWNER,I like this. It could work with the `--root` option too and automatically sign you in as the root user.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879531,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695879531,MDEyOklzc3VlQ29tbWVudDY5NTg3OTUzMQ==,9599,2020-09-21T02:55:28Z,2020-09-21T02:55:54Z,MEMBER,"Actually for the tie-breaker it should be something like https://latest.datasette.io/fixtures?sql=select+pk%2C+created%2C+planet_int%2C+on_earth%2C+state%2C+city_id%2C+neighborhood%2C+tags%2C+complex_array%2C+distinct_some_null+from+facetable+where+%28created+%3E+%3Ap1+or+%28created+%3D+%3Ap1+and+%28%28pk+%3E+%3Ap0%29%29%29%29+order+by+created%2C+pk+limit+11&p0=10&p1=2019-01-16+08%3A00%3A00 ```sql where ( created > :p1 or ( created = :p1 and ((pk > :p0)) ) ) ``` But with `rowid` and `timestamp` in place of `pk` and `created`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879237,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695879237,MDEyOklzc3VlQ29tbWVudDY5NTg3OTIzNw==,9599,2020-09-21T02:53:29Z,2020-09-21T02:53:29Z,MEMBER,"If previous page ended at `2018-02-11T16:32:53+00:00`: ```sql select search_index.rowid, search_index.type, search_index.key, search_index.title, search_index.category, search_index.timestamp, search_index.search_1 from search_index where date(""timestamp"") = '2018-02-11' and timestamp < '2018-02-11T16:32:53+00:00' order by search_index.timestamp desc, rowid limit 41 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230, https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695877627,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16,695877627,MDEyOklzc3VlQ29tbWVudDY5NTg3NzYyNw==,9599,2020-09-21T02:42:29Z,2020-09-21T02:42:29Z,MEMBER,"Fun twist: assuming `timestamp` is always stored as UTC, I need the interface to be timezone aware so I can see e.g. everything from 4th July 2020 in the San Francisco timezone definition of 4th July 2020.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694493566, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695875274,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695875274,MDEyOklzc3VlQ29tbWVudDY5NTg3NTI3NA==,9599,2020-09-21T02:28:58Z,2020-09-21T02:28:58Z,MEMBER,Datasette's implementation is complex because it has to support compound primary keys: https://github.com/simonw/datasette/blob/a258339a935d8d29a95940ef1db01e98bb85ae63/datasette/utils/__init__.py#L88-L114 - but that's not something that's needed for dogsheep-beta.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856967,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695856967,MDEyOklzc3VlQ29tbWVudDY5NTg1Njk2Nw==,9599,2020-09-21T00:26:59Z,2020-09-21T00:26:59Z,MEMBER,It's a shame Datasette doesn't currently have an easy way to implement sorted-by-rank keyset-paginated using a TableView or QueryView. I'll have to do this using the custom SQL query constructed in the plugin: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L8-L43,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856398,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695856398,MDEyOklzc3VlQ29tbWVudDY5NTg1NjM5OA==,9599,2020-09-21T00:22:20Z,2020-09-21T00:22:20Z,MEMBER,I'm going to try for keyset pagination sorted by relevance just as a learning exercise.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855723,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695855723,MDEyOklzc3VlQ29tbWVudDY5NTg1NTcyMw==,9599,2020-09-21T00:16:52Z,2020-09-21T00:17:53Z,MEMBER,"It feels a bit weird to implement keyset pagination against results sorted by `rank` because the ranks could change substantially if the search index gets updated while the user is paginating. I may just ignore that though. If you want reliable pagination you can get it by sorting by date. Maybe it doesn't even make sense to offer pagination if you sort by relevance?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855646,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695855646,MDEyOklzc3VlQ29tbWVudDY5NTg1NTY0Ng==,9599,2020-09-21T00:16:11Z,2020-09-21T00:16:11Z,MEMBER,"Should I do this with offset/limit or should I do proper keyset pagination? I think keyset because then it will work well for the full search interface with no filters or search string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230, https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695851036,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16,695851036,MDEyOklzc3VlQ29tbWVudDY5NTg1MTAzNg==,9599,2020-09-20T23:34:57Z,2020-09-20T23:34:57Z,MEMBER,Really basic starting point is to add facet by date.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694493566, https://github.com/simonw/sqlite-utils/issues/160#issuecomment-695839557,https://api.github.com/repos/simonw/sqlite-utils/issues/160,695839557,MDEyOklzc3VlQ29tbWVudDY5NTgzOTU1Nw==,9599,2020-09-20T21:37:03Z,2020-09-20T21:37:03Z,OWNER,"Should this support `ignore=True` as well? I'm tempted to skip that - I think `replace=True` is more useful because it implies ""ignore if the options are already the same, but replace if they are different"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705190723, https://github.com/simonw/sqlite-utils/issues/42#issuecomment-695698227,https://api.github.com/repos/simonw/sqlite-utils/issues/42,695698227,MDEyOklzc3VlQ29tbWVudDY5NTY5ODIyNw==,9599,2020-09-20T04:27:26Z,2020-09-20T04:28:26Z,OWNER,This is going to need #114 (the `transform_table()` method) in order to convert string columns into integer foreign key columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929, https://github.com/simonw/sqlite-utils/issues/68#issuecomment-695695776,https://api.github.com/repos/simonw/sqlite-utils/issues/68,695695776,MDEyOklzc3VlQ29tbWVudDY5NTY5NTc3Ng==,9599,2020-09-20T04:25:47Z,2020-09-20T04:25:47Z,OWNER,This is a dupe of #130 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531583658, https://github.com/simonw/datasette/issues/943#issuecomment-695133768,https://api.github.com/repos/simonw/datasette/issues/943,695133768,MDEyOklzc3VlQ29tbWVudDY5NTEzMzc2OA==,9599,2020-09-19T00:06:56Z,2020-09-19T00:07:35Z,OWNER,"[dogsheep-beta](https://github.com/dogsheep/dogsheep-beta) could do with this too. It currently [makes a call](https://github.com/dogsheep/dogsheep-beta/blob/ab36101bdae69b11af7c6bd7edee838d052e6ecf/dogsheep_beta/__init__.py#L216-L225) to `TableView` in a similar way to `datasette-graphql` in order to calculate facets. `dogsheep-beta` would benefit with a mechanism for changing the facet timeout setting during that call (as would `datasette-graphql`, see the [DatasetteSpecialConfig mechanism](https://github.com/simonw/datasette-graphql/blob/f9dc5c518b7cdc94b93873ef20069a7ea2882a95/datasette_graphql/utils.py#L516-L519) it uses).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/dogsheep/dogsheep-beta/issues/15#issuecomment-695124698,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15,695124698,MDEyOklzc3VlQ29tbWVudDY5NTEyNDY5OA==,9599,2020-09-18T23:17:38Z,2020-09-18T23:17:38Z,MEMBER,This can be part of the demo instance in #6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694136490, https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-695113871,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,695113871,MDEyOklzc3VlQ29tbWVudDY5NTExMzg3MQ==,9599,2020-09-18T22:30:17Z,2020-09-18T22:30:17Z,MEMBER,"I think I know what's going on here: https://github.com/dogsheep/dogsheep-beta/blob/0f1b951c5131d16f3c8559a8e4d79ed5c559e3cb/dogsheep_beta/__init__.py#L166-L171 This is a logic bug - the `compiled` variable could be the template from the previous loop!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814, https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695109140,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25,695109140,MDEyOklzc3VlQ29tbWVudDY5NTEwOTE0MA==,9599,2020-09-18T22:12:20Z,2020-09-18T22:12:20Z,MEMBER,Documented here: https://github.com/dogsheep/dogsheep-beta/blob/534fc9689227eba70e69a45da0cee5820bbda9e1/README.md#datasette-plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",704685890, https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695108895,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25,695108895,MDEyOklzc3VlQ29tbWVudDY5NTEwODg5NQ==,9599,2020-09-18T22:11:32Z,2020-09-18T22:11:32Z,MEMBER,"I'm going to make this a new plugin configuration setting, `template_debug`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",704685890, https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694557425,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694557425,MDEyOklzc3VlQ29tbWVudDY5NDU1NzQyNQ==,9599,2020-09-17T23:41:01Z,2020-09-17T23:41:01Z,MEMBER,I removed all of the `json.loads()` calls and I'm still getting that `Undefined` error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814, https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694554584,MDEyOklzc3VlQ29tbWVudDY5NDU1NDU4NA==,9599,2020-09-17T23:31:25Z,2020-09-17T23:31:25Z,MEMBER,"I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814, https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694553579,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694553579,MDEyOklzc3VlQ29tbWVudDY5NDU1MzU3OQ==,9599,2020-09-17T23:28:37Z,2020-09-17T23:28:37Z,MEMBER,"More investigation in pdb: ``` (dogsheep-beta) dogsheep-beta % datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb > /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads() -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' (Pdb) list 336 if s.startswith('\ufeff'): 337 raise JSONDecodeError(""Unexpected UTF-8 BOM (decode using utf-8-sig)"", 338 s, 0) 339 else: 340 if not isinstance(s, (bytes, bytearray)): 341 -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' 342 f'not {s.__class__.__name__}') 343 s = s.decode(detect_encoding(s), 'surrogatepass') 344 345 if ""encoding"" in kw: 346 import warnings (Pdb) bytes (Pdb) locals()['s'] Undefined (Pdb) type(locals()['s']) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814, https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552681,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694552681,MDEyOklzc3VlQ29tbWVudDY5NDU1MjY4MQ==,9599,2020-09-17T23:25:54Z,2020-09-17T23:25:54Z,MEMBER,"This is the template fragment it's rendering: ```html+jinja

Tweet by @{{ display.screen_name }} ({{ display.user_name }}, {{ ""{:,}"".format(display.followers_count or 0) }} followers) on {{ display.created_at }}

{{ display.full_text }}
{% if display.media_urls and json.loads(display.media_urls) %} {% for url in json.loads(display.media_urls) %} {% endfor %} {% endif %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814, https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552393,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694552393,MDEyOklzc3VlQ29tbWVudDY5NDU1MjM5Mw==,9599,2020-09-17T23:25:01Z,2020-09-17T23:25:17Z,MEMBER,"Ran `locals()` In the debugger: `{'range': , 'dict': , 'lipsum': , 'cycler': , 'joiner': , 'namespace': , 'rank': -9.383801886431414, 'rowid': 14297, 'type': 'twitter.db/tweets', 'key': '312658917933076480', 'title': 'Tweet by @chrisstreeter', 'category': 2, 'timestamp': '2013-03-15T20:17:49+00:00', 'search_1': '@simonw are you at pycon? Would love to meet you.', 'display': {'avatar_url': 'https://pbs.twimg.com/profile_images/806275088597204993/38yLHfJi_normal.jpg', 'user_name': 'Chris Streeter', 'screen_name': 'chrisstreeter', 'followers_count': 280, 'tweet_id': 312658917933076480, 'created_at': '2013-03-15T20:17:49+00:00', 'full_text': '@simonw are you at pycon? Would love to meet you.', 'media_urls_2': '[]', 'media_urls': '[]'}, 'json': }`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814, https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551646,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694551646,MDEyOklzc3VlQ29tbWVudDY5NDU1MTY0Ng==,9599,2020-09-17T23:22:48Z,2020-09-17T23:22:48Z,MEMBER,"Looks like its happening in a Jinja fragment template for one of the results: ``` /Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/__init__.py(169)process_results() -> output = compiled.render({**result, **{""json"": json}}) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/asyncsupport.py(71)render() -> return original_render(self, *args, **kwargs) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(1090)render() -> self.environment.handle_exception() /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(832)handle_exception() -> reraise(*rewrite_traceback_stack(source=source)) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/_compat.py(28)reraise() -> raise value.with_traceback(tb)