html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073448904,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073448904,IC_kwDOCGYnMM4_-4fI,9599,2022-03-21T03:28:12Z,2022-03-21T03:30:37Z,OWNER,"Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/
```
sqlite-utils create-database test-dates.db
sqlite-utils create-table test-dates.db dates id integer date text --pk id
sqlite-utils test-dates.db ""WITH RECURSIVE
cnt(x) AS (
SELECT 0
UNION ALL
SELECT x+1 FROM cnt
LIMIT (SELECT ((julianday('2016-04-01') - julianday('2016-03-15'))) + 1)
)
insert into dates (date) select date(julianday('2016-03-15'), '+' || x || ' days') as date FROM cnt;""
```
After running that:
```
% sqlite-utils rows test-dates.db dates
[{""id"": 1, ""date"": ""2016-03-15""},
{""id"": 2, ""date"": ""2016-03-16""},
{""id"": 3, ""date"": ""2016-03-17""},
{""id"": 4, ""date"": ""2016-03-18""},
{""id"": 5, ""date"": ""2016-03-19""},
{""id"": 6, ""date"": ""2016-03-20""},
{""id"": 7, ""date"": ""2016-03-21""},
{""id"": 8, ""date"": ""2016-03-22""},
{""id"": 9, ""date"": ""2016-03-23""},
{""id"": 10, ""date"": ""2016-03-24""},
{""id"": 11, ""date"": ""2016-03-25""},
{""id"": 12, ""date"": ""2016-03-26""},
{""id"": 13, ""date"": ""2016-03-27""},
{""id"": 14, ""date"": ""2016-03-28""},
{""id"": 15, ""date"": ""2016-03-29""},
{""id"": 16, ""date"": ""2016-03-30""},
{""id"": 17, ""date"": ""2016-03-31""},
{""id"": 18, ""date"": ""2016-04-01""}]
```
Then to make one of them invalid:
sqlite-utils test-dates.db ""update dates set date = '//' where id = 10""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073450588,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073450588,IC_kwDOCGYnMM4_-45c,9599,2022-03-21T03:32:58Z,2022-03-21T03:32:58Z,OWNER,"Then I ran this to convert `2016-03-27` etc to `2016/03/27` so I could see which ones were later converted:
sqlite-utils convert test-dates.db dates date 'value.replace(""-"", ""/"")'
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073451659,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073451659,IC_kwDOCGYnMM4_-5KL,9599,2022-03-21T03:35:01Z,2022-03-21T03:35:01Z,OWNER,"I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction.
Here's the code that does that:
https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L2523-L2526","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453230,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073453230,IC_kwDOCGYnMM4_-5iu,9599,2022-03-21T03:40:37Z,2022-03-21T03:40:37Z,OWNER,"I think the options here should be:
- On error, raise an exception and revert the transaction (the current default)
- On error, leave the value as-is
- On error, set the value to `None`
These need to be indicated by parameters to the `r.parsedate()` function.
Some design options:
- `ignore=True` to ignore errors - but how does it know if it should leave the value or set it to `None`? This is similar to other `ignore=True` parameters elsewhere in the Python API.
- `errors=""ignore""`, `errors=""set-null""` - I don't like magic string values very much, but this is similar to Python's `str.encode(errors=)` mechanism
- `errors=r.IGNORE` - using constants, which at least avoids magic strings. The other one could be `errors=r.SET_NULL`
- `error=lambda v: None` or `error=lambda v: v` - this is a bit confusing though, introducing another callback that gets to have a go at converting the error if the first callback failed? And what happens if that lambda itself raises an error?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453370,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073453370,IC_kwDOCGYnMM4_-5k6,9599,2022-03-21T03:41:06Z,2022-03-21T03:41:06Z,OWNER,I'm going to try the `errors=r.IGNORE` option and see what that looks like once implemented.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073455905,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073455905,IC_kwDOCGYnMM4_-6Mh,9599,2022-03-21T03:44:47Z,2022-03-21T03:45:00Z,OWNER,"This is quite nice:
```
% sqlite-utils convert test-dates.db dates date ""r.parsedate(value, errors=r.IGNORE)""
[####################################] 100%
% sqlite-utils rows test-dates.db dates
[{""id"": 1, ""date"": ""2016-03-15""},
{""id"": 2, ""date"": ""2016-03-16""},
{""id"": 3, ""date"": ""2016-03-17""},
{""id"": 4, ""date"": ""2016-03-18""},
{""id"": 5, ""date"": ""2016-03-19""},
{""id"": 6, ""date"": ""2016-03-20""},
{""id"": 7, ""date"": ""2016-03-21""},
{""id"": 8, ""date"": ""2016-03-22""},
{""id"": 9, ""date"": ""2016-03-23""},
{""id"": 10, ""date"": ""//""},
{""id"": 11, ""date"": ""2016-03-25""},
{""id"": 12, ""date"": ""2016-03-26""},
{""id"": 13, ""date"": ""2016-03-27""},
{""id"": 14, ""date"": ""2016-03-28""},
{""id"": 15, ""date"": ""2016-03-29""},
{""id"": 16, ""date"": ""2016-03-30""},
{""id"": 17, ""date"": ""2016-03-31""},
{""id"": 18, ""date"": ""2016-04-01""}]
% sqlite-utils convert test-dates.db dates date ""r.parsedate(value, errors=r.SET_NULL)""
[####################################] 100%
% sqlite-utils rows test-dates.db dates
[{""id"": 1, ""date"": ""2016-03-15""},
{""id"": 2, ""date"": ""2016-03-16""},
{""id"": 3, ""date"": ""2016-03-17""},
{""id"": 4, ""date"": ""2016-03-18""},
{""id"": 5, ""date"": ""2016-03-19""},
{""id"": 6, ""date"": ""2016-03-20""},
{""id"": 7, ""date"": ""2016-03-21""},
{""id"": 8, ""date"": ""2016-03-22""},
{""id"": 9, ""date"": ""2016-03-23""},
{""id"": 10, ""date"": null},
{""id"": 11, ""date"": ""2016-03-25""},
{""id"": 12, ""date"": ""2016-03-26""},
{""id"": 13, ""date"": ""2016-03-27""},
{""id"": 14, ""date"": ""2016-03-28""},
{""id"": 15, ""date"": ""2016-03-29""},
{""id"": 16, ""date"": ""2016-03-30""},
{""id"": 17, ""date"": ""2016-03-31""},
{""id"": 18, ""date"": ""2016-04-01""}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456155,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073456155,IC_kwDOCGYnMM4_-6Qb,9599,2022-03-21T03:45:37Z,2022-03-21T03:45:37Z,OWNER,"Prototype:
```diff
diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
index 8255b56..0a3693e 100644
--- a/sqlite_utils/cli.py
+++ b/sqlite_utils/cli.py
@@ -2583,7 +2583,11 @@ def _generate_convert_help():
""""""
).strip()
recipe_names = [
- n for n in dir(recipes) if not n.startswith(""_"") and n not in (""json"", ""parser"")
+ n
+ for n in dir(recipes)
+ if not n.startswith(""_"")
+ and n not in (""json"", ""parser"")
+ and callable(getattr(recipes, n))
]
for name in recipe_names:
fn = getattr(recipes, name)
diff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py
index 6918661..569c30d 100644
--- a/sqlite_utils/recipes.py
+++ b/sqlite_utils/recipes.py
@@ -1,17 +1,38 @@
from dateutil import parser
import json
+IGNORE = object()
+SET_NULL = object()
-def parsedate(value, dayfirst=False, yearfirst=False):
+
+def parsedate(value, dayfirst=False, yearfirst=False, errors=None):
""Parse a date and convert it to ISO date format: yyyy-mm-dd""
- return (
- parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat()
- )
+ try:
+ return (
+ parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst)
+ .date()
+ .isoformat()
+ )
+ except parser.ParserError:
+ if errors is IGNORE:
+ return value
+ elif errors is SET_NULL:
+ return None
+ else:
+ raise
-def parsedatetime(value, dayfirst=False, yearfirst=False):
+def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None):
""Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS""
- return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat()
+ try:
+ return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat()
+ except parser.ParserError:
+ if errors is IGNORE:
+ return value
+ elif errors is SET_NULL:
+ return None
+ else:
+ raise
def jsonsplit(value, delimiter="","", type=str):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456222,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073456222,IC_kwDOCGYnMM4_-6Re,9599,2022-03-21T03:45:52Z,2022-03-21T03:45:52Z,OWNER,Needs tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,
https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073463375,https://api.github.com/repos/simonw/sqlite-utils/issues/415,1073463375,IC_kwDOCGYnMM4_-8BP,9599,2022-03-21T04:02:36Z,2022-03-21T04:02:36Z,OWNER,Thanks for the really clear steps to reproduce!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1171599874,
https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073468996,https://api.github.com/repos/simonw/sqlite-utils/issues/415,1073468996,IC_kwDOCGYnMM4_-9ZE,9599,2022-03-21T04:14:42Z,2022-03-21T04:14:42Z,OWNER,"I can fix this like so:
```
% sqlite-utils convert demo.db demo foo '{""foo"": ""bar""}' --multi --dry-run
abc
--- becomes:
{""foo"": ""bar""}
Would affect 1 row
```
Diff is this:
```diff
diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
index 0cf0468..b2a0440 100644
--- a/sqlite_utils/cli.py
+++ b/sqlite_utils/cli.py
@@ -2676,7 +2676,10 @@ def convert(
raise click.ClickException(str(e))
if dry_run:
# Pull first 20 values for first column and preview them
- db.conn.create_function(""preview_transform"", 1, lambda v: fn(v) if v else v)
+ preview = lambda v: fn(v) if v else v
+ if multi:
+ preview = lambda v: json.dumps(fn(v), default=repr) if v else v
+ db.conn.create_function(""preview_transform"", 1, preview)
sql = """"""
select
[{column}] as value,
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1171599874,
https://github.com/simonw/datasette/issues/1177#issuecomment-1074017633,https://api.github.com/repos/simonw/datasette/issues/1177,1074017633,IC_kwDOBm6k_c5ABDVh,9599,2022-03-21T15:08:51Z,2022-03-21T15:08:51Z,OWNER,"Related:
- #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780153562,
https://github.com/simonw/datasette/issues/526#issuecomment-1074019047,https://api.github.com/repos/simonw/datasette/issues/526,1074019047,IC_kwDOBm6k_c5ABDrn,9599,2022-03-21T15:09:56Z,2022-03-21T15:09:56Z,OWNER,I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,
https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176,https://api.github.com/repos/simonw/datasette/issues/1660,1074136176,IC_kwDOBm6k_c5ABgRw,9599,2022-03-21T16:38:46Z,2022-03-21T16:38:46Z,OWNER,"I'm going to refactor this stuff out and document it so it can be easily used by plugins:
https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074141457,https://api.github.com/repos/simonw/datasette/issues/1675,1074141457,IC_kwDOBm6k_c5ABhkR,9599,2022-03-21T16:44:09Z,2022-03-21T16:44:09Z,OWNER,"A slightly odd thing about these methods is that they either fail silently or they raise a `Forbidden` exception.
Maybe they should instead return `True` or `False` and the calling code could decide if it wants to raise the exception? That would make them more usable and a little less surprising.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074142617,https://api.github.com/repos/simonw/datasette/issues/1675,1074142617,IC_kwDOBm6k_c5ABh2Z,9599,2022-03-21T16:45:27Z,2022-03-21T16:45:27Z,OWNER,"Though at that point `check_permission` is such a light wrapper around `self.ds.permission_allowed()` that there's little point in it existing at all.
So maybe `check_permisions()` becomes `ds.permissions_allowed()`.
`permission_allowed()` v.s. `permissions_allowed()` is a bit of a subtle naming difference, but I think it works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074143209,https://api.github.com/repos/simonw/datasette/issues/1675,1074143209,IC_kwDOBm6k_c5ABh_p,9599,2022-03-21T16:46:05Z,2022-03-21T16:46:05Z,OWNER,"The other difference though is that `ds.permission_allowed(...)` works against an actor, while `check_permission()` works against a request (though just to access `request.actor`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074156779,https://api.github.com/repos/simonw/datasette/issues/1675,1074156779,IC_kwDOBm6k_c5ABlTr,9599,2022-03-21T16:55:08Z,2022-03-21T16:56:02Z,OWNER,"One benefit of the current design of `check_permissions` that raises an exception is that the exception includes information on WHICH of the permission checks failed. Returning just `True` or `False` loses that information.
I could return an object which evaluates to `False` but also carries extra information? Bit weird, I've never seen anything like that in other Python code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074158890,https://api.github.com/repos/simonw/datasette/issues/1675,1074158890,IC_kwDOBm6k_c5ABl0q,9599,2022-03-21T16:57:15Z,2022-03-21T16:57:15Z,OWNER,"Idea: `ds.permission_allowed()` continues to just return `True` or `False`.
A new `ds.ensure_permissions(...)` method is added which raises a `Forbidden` exception if a check fails (hence the different name)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074161523,https://api.github.com/repos/simonw/datasette/issues/1675,1074161523,IC_kwDOBm6k_c5ABmdz,9599,2022-03-21T16:59:55Z,2022-03-21T17:00:03Z,OWNER,Also calling that function `permissions_allowed()` is confusing because there is a plugin hook with a similar name already: https://docs.datasette.io/en/stable/plugin_hooks.html#permission-allowed-datasette-actor-action-resource,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827,https://api.github.com/repos/simonw/datasette/issues/1675,1074177827,IC_kwDOBm6k_c5ABqcj,9599,2022-03-21T17:14:31Z,2022-03-21T17:14:31Z,OWNER,"Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions
> This method allows multiple permissions to be checked at onced. It raises a `datasette.Forbidden` exception if any of the checks are denied before one of them is explicitly granted.
>
> This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns `True` or not a single one of them returns `False`:
That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1676#issuecomment-1074178865,https://api.github.com/repos/simonw/datasette/issues/1676,1074178865,IC_kwDOBm6k_c5ABqsx,9599,2022-03-21T17:15:27Z,2022-03-21T17:15:27Z,OWNER,This method here: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/app.py#L632-L664,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,
https://github.com/simonw/datasette/issues/1676#issuecomment-1074180312,https://api.github.com/repos/simonw/datasette/issues/1676,1074180312,IC_kwDOBm6k_c5ABrDY,9599,2022-03-21T17:16:45Z,2022-03-21T17:16:45Z,OWNER,"When looking at this code earlier I assumed that the following would check each permission in turn and fail if any of them failed:
```python
await self.ds.ensure_permissions(
request.actor,
[
(""view-table"", (database, table)),
(""view-database"", database),
""view-instance"",
]
)
```
But it's not quite that simple: if any of them fail, it fails... but if an earlier one returns `True` the whole stack passes even if there would have been a failure later on!
If that is indeed the right abstraction, I need to work to make the documentation as clear as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,
https://github.com/simonw/datasette/issues/1677#issuecomment-1074184240,https://api.github.com/repos/simonw/datasette/issues/1677,1074184240,IC_kwDOBm6k_c5ABsAw,9599,2022-03-21T17:20:17Z,2022-03-21T17:20:17Z,OWNER,"https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/views/base.py#L69-L77
This is weirdly different from how `check_permissions()` used to work, in that it doesn't differentiate between `None` and `False`.
https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L79-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175694248,
https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074243540,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1074243540,IC_kwDOCGYnMM5AB6fU,9599,2022-03-21T18:08:03Z,2022-03-21T18:08:03Z,OWNER,"I've not really thought about standards as much here as I should. It looks like there are two competing specs for newline-delimited JSON!
http://ndjson.org/ is the one I've been using in `sqlite-utils` - and https://github.com/ndjson/ndjson-spec#31-serialization says:
> The JSON texts MUST NOT contain newlines or carriage returns.
https://jsonlines.org/ is the other one. It is slightly less clear, but it does say this:
> 2. Each Line is a Valid JSON Value
>
> The most common values will be objects or arrays, but any JSON value is permitted.
My interpretation of both of these is that newlines in the middle of a JSON object shouldn't be allowed.
So what's `jq` doing here? It looks to me like that `jq` format is its own thing - it's not actually compatible with either of those two loose specs described above.
The `jq` docs seem to call this ""whitespace-separated JSON"": https://stedolan.github.io/jq/manual/v1.6/#Invokingjq
The thing I like about newline-delimited JSON is that it's really trivial to parse - loop through each line, run it through `json.loads()` and that's it. No need to try and unwrap JSON objects that might span multiple lines.
Unless someone has written a robust Python implementation of a `jq`-compatible whitespace-separated JSON parser, I'm inclined to leave this as is. I'd be fine adding some documentation that helps point people towards `jq -c` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,
https://github.com/simonw/datasette/issues/1660#issuecomment-1074287177,https://api.github.com/repos/simonw/datasette/issues/1660,1074287177,IC_kwDOBm6k_c5ACFJJ,9599,2022-03-21T18:51:42Z,2022-03-21T18:51:42Z,OWNER,`BaseView` is looking a LOT slimmer now that I've moved all of the permissions stuff out of it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,
https://github.com/simonw/datasette/issues/1678#issuecomment-1074302559,https://api.github.com/repos/simonw/datasette/issues/1678,1074302559,IC_kwDOBm6k_c5ACI5f,9599,2022-03-21T19:04:03Z,2022-03-21T19:04:03Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-resource-none,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175715988,
https://github.com/simonw/datasette/issues/1660#issuecomment-1074321862,https://api.github.com/repos/simonw/datasette/issues/1660,1074321862,IC_kwDOBm6k_c5ACNnG,9599,2022-03-21T19:19:01Z,2022-03-21T19:19:01Z,OWNER,I've simplified this a ton now. I'm going to keep working on this in the long-term but I think this issue can be closed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074331743,https://api.github.com/repos/simonw/datasette/issues/1679,1074331743,IC_kwDOBm6k_c5ACQBf,9599,2022-03-21T19:30:05Z,2022-03-21T19:30:05Z,OWNER,"https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/app.py#L118-L122 sets it to 50ms for facet suggestion but that's not going to pass `ms < 50`:
```python
Setting(
""facet_suggest_time_limit_ms"",
50,
""Time limit for calculating a suggested facet"",
),
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074332325,https://api.github.com/repos/simonw/datasette/issues/1679,1074332325,IC_kwDOBm6k_c5ACQKl,9599,2022-03-21T19:30:44Z,2022-03-21T19:30:44Z,OWNER,So it looks like even for facet suggestion `n=1000` always - it's never reduced to `n=1`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074332718,https://api.github.com/repos/simonw/datasette/issues/1679,1074332718,IC_kwDOBm6k_c5ACQQu,9599,2022-03-21T19:31:10Z,2022-03-21T19:31:10Z,OWNER,How long does it take for SQLite to execute 1000 opcodes anyway?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074337997,https://api.github.com/repos/simonw/datasette/issues/1679,1074337997,IC_kwDOBm6k_c5ACRjN,9599,2022-03-21T19:37:08Z,2022-03-21T19:37:08Z,OWNER,"This is weird:
```python
import sqlite3
db = sqlite3.connect("":memory:"")
i = 0
def count():
global i
i += 1
db.set_progress_handler(count, 1)
db.execute(""""""
with recursive counter(x) as (
select 0
union
select x + 1 from counter
)
select * from counter limit 10000;
"""""")
print(i)
```
Outputs `24`. But if you try the same thing in the SQLite console:
```
sqlite> .stats vmstep
sqlite> with recursive counter(x) as (
...> select 0
...> union
...> select x + 1 from counter
...> )
...> select * from counter limit 10000;
...
VM-steps: 200007
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074341924,https://api.github.com/repos/simonw/datasette/issues/1679,1074341924,IC_kwDOBm6k_c5ACSgk,9599,2022-03-21T19:42:08Z,2022-03-21T19:42:08Z,OWNER,"Here's the Python-C implementation of `set_progress_handler`: https://github.com/python/cpython/blob/4674fd4e938eb4a29ccd5b12c15455bd2a41c335/Modules/_sqlite/connection.c#L1177-L1201
It calls `sqlite3_progress_handler(self->db, n, progress_callback, ctx);`
https://www.sqlite.org/c3ref/progress_handler.html says:
> The parameter N is the approximate number of [virtual machine instructions](https://www.sqlite.org/opcode.html) that are evaluated between successive invocations of the callback X
So maybe VM-steps and virtual machine instructions are different things?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074347023,https://api.github.com/repos/simonw/datasette/issues/1679,1074347023,IC_kwDOBm6k_c5ACTwP,9599,2022-03-21T19:48:59Z,2022-03-21T19:48:59Z,OWNER,Posed a question about that here: https://sqlite.org/forum/forumpost/de9ff10fa7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1676#issuecomment-1074378472,https://api.github.com/repos/simonw/datasette/issues/1676,1074378472,IC_kwDOBm6k_c5ACbbo,9599,2022-03-21T20:18:10Z,2022-03-21T20:18:10Z,OWNER,Maybe there is a better name for this method that helps emphasize its cascading nature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074439309,https://api.github.com/repos/simonw/datasette/issues/1679,1074439309,IC_kwDOBm6k_c5ACqSN,9599,2022-03-21T21:28:58Z,2022-03-21T21:28:58Z,OWNER,"David Raymond solved it there: https://sqlite.org/forum/forumpost/330c8532d8a88bcd
> Don't forget to step through the results. All .execute() has done is prepared it.
>
> db.execute(query).fetchall()
Sure enough, adding that gets the VM steps number up to 190,007 which is close enough that I'm happy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074446576,https://api.github.com/repos/simonw/datasette/issues/1679,1074446576,IC_kwDOBm6k_c5ACsDw,9599,2022-03-21T21:38:27Z,2022-03-21T21:38:27Z,OWNER,"OK here's a microbenchmark script:
```python
import sqlite3
import timeit
db = sqlite3.connect("":memory:"")
db_with_progress_handler_1 = sqlite3.connect("":memory:"")
db_with_progress_handler_1000 = sqlite3.connect("":memory:"")
db_with_progress_handler_1.set_progress_handler(lambda: None, 1)
db_with_progress_handler_1000.set_progress_handler(lambda: None, 1000)
def execute_query(db):
cursor = db.execute(""""""
with recursive counter(x) as (
select 0
union
select x + 1 from counter
)
select * from counter limit 10000;
"""""")
list(cursor.fetchall())
print(""Without progress_handler"")
print(timeit.timeit(lambda: execute_query(db), number=100))
print(""progress_handler every 1000 ops"")
print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1000), number=100))
print(""progress_handler every 1 op"")
print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1), number=100))
```
Results:
```
% python3 bench.py
Without progress_handler
0.8789225700311363
progress_handler every 1000 ops
0.8829826560104266
progress_handler every 1 op
2.8892734259716235
```
So running every 1000 ops makes almost no difference at all, but running every single op is a 3.2x performance degradation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074454687,https://api.github.com/repos/simonw/datasette/issues/1679,1074454687,IC_kwDOBm6k_c5ACuCf,9599,2022-03-21T21:48:02Z,2022-03-21T21:48:02Z,OWNER,"Here's another microbenchmark that measures how many nanoseconds it takes to run 1,000 vmops:
```python
import sqlite3
import time
db = sqlite3.connect("":memory:"")
i = 0
out = []
def count():
global i
i += 1000
out.append(((i, time.perf_counter_ns())))
db.set_progress_handler(count, 1000)
print(""Start:"", time.perf_counter_ns())
all = db.execute(""""""
with recursive counter(x) as (
select 0
union
select x + 1 from counter
)
select * from counter limit 10000;
"""""").fetchall()
print(""End:"", time.perf_counter_ns())
print()
print(""So how long does it take to execute 1000 ops?"")
prev_time_ns = None
for i, time_ns in out:
if prev_time_ns is not None:
print(time_ns - prev_time_ns, ""ns"")
prev_time_ns = time_ns
```
Running it:
```
% python nanobench.py
Start: 330877620374821
End: 330877632515822
So how long does it take to execute 1000 ops?
47290 ns
49573 ns
48226 ns
45674 ns
53238 ns
47313 ns
52346 ns
48689 ns
47092 ns
87596 ns
69999 ns
52522 ns
52809 ns
53259 ns
52478 ns
53478 ns
65812 ns
```
87596ns is 0.087596ms - so even a measure rate of every 1000 ops is easily finely grained enough to capture differences of less than 0.1ms.
If anything I could bump that default 1000 up - and I can definitely eliminate the `if ms < 50` branch entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074458506,https://api.github.com/repos/simonw/datasette/issues/1679,1074458506,IC_kwDOBm6k_c5ACu-K,9599,2022-03-21T21:53:47Z,2022-03-21T21:53:47Z,OWNER,"Oh interesting, it turns out there is ONE place in the code that sets the `ms` to less than 20 - this test fixture: https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/tests/fixtures.py#L224-L226","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074459746,https://api.github.com/repos/simonw/datasette/issues/1679,1074459746,IC_kwDOBm6k_c5ACvRi,9599,2022-03-21T21:55:45Z,2022-03-21T21:55:45Z,OWNER,I'm going to change the original logic to set n=1 for times that are `<= 20ms` - and update the comments to make it more obvious what is happening.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1671#issuecomment-1074465536,https://api.github.com/repos/simonw/datasette/issues/1671,1074465536,IC_kwDOBm6k_c5ACwsA,9599,2022-03-21T22:04:31Z,2022-03-21T22:04:31Z,OWNER,"Oh this is fascinating! I replicated the bug (thanks for the steps to reproduce) and it looks like this is down to the following:
Against views, `where has_expired = 1` returns different results from `where has_expired = '1'`
This doesn't happen against tables because of SQLite's [type affinity](https://www.sqlite.org/datatype3.html#type_affinity) mechanism, which handles the type conversion automatically.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/1671#issuecomment-1074468450,https://api.github.com/repos/simonw/datasette/issues/1671,1074468450,IC_kwDOBm6k_c5ACxZi,9599,2022-03-21T22:08:35Z,2022-03-21T22:10:00Z,OWNER,"Relevant section of the SQLite documentation: [3.2. Affinity Of Expressions](https://www.sqlite.org/datatype3.html#affinity_of_expressions):
> When an expression is a simple reference to a column of a real table (not a [VIEW](https://www.sqlite.org/lang_createview.html) or subquery) then the expression has the same affinity as the table column.
In your example, `has_expired` is no longer a simple reference to a column of a real table, hence the bug.
Then [4.2. Type Conversions Prior To Comparison](https://www.sqlite.org/datatype3.html#type_conversions_prior_to_comparison) fills in the rest:
> SQLite may attempt to convert values between the storage classes INTEGER, REAL, and/or TEXT before performing a comparison. Whether or not any conversions are attempted before the comparison takes place depends on the type affinity of the operands. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/1671#issuecomment-1074470568,https://api.github.com/repos/simonw/datasette/issues/1671,1074470568,IC_kwDOBm6k_c5ACx6o,9599,2022-03-21T22:11:14Z,2022-03-21T22:12:49Z,OWNER,"I wonder if this will be a problem with generated columns, or with SQLite strict tables?
My hunch is that strict tables will continue to work without any changes, because https://www.sqlite.org/stricttables.html says nothing about their impact on comparison operations. I should test this to make absolutely sure though.
Generated columns have a type, so my hunch is they will continue to work fine too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/1671#issuecomment-1074478299,https://api.github.com/repos/simonw/datasette/issues/1671,1074478299,IC_kwDOBm6k_c5ACzzb,9599,2022-03-21T22:20:26Z,2022-03-21T22:20:26Z,OWNER,"Thinking about options for fixing this...
The following query works fine:
```sql
select * from test_view where cast(has_expired as text) = '1'
```
I don't want to start using this for every query, because one of the goals of Datasette is to help people who are learning SQL:
- #1613
If someone clicks on ""View and edit SQL"" from a filtered table page I don't want them to have to wonder why that `cast` is there.
But... for querying views, the `cast` turns out to be necessary.
So one fix would be to get the SQL generating logic to use casts like this any time it is operating against a view.
An even better fix would be to detect which columns in a view come from a table and which ones might not, and only use casts for the columns that aren't definitely from a table.
The trick I was exploring here might be able to help with that:
- #1293 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/276#issuecomment-1074479768,https://api.github.com/repos/simonw/datasette/issues/276,1074479768,IC_kwDOBm6k_c5AC0KY,9599,2022-03-21T22:22:20Z,2022-03-21T22:22:20Z,OWNER,"I'm closing this issue because this is now solved by a number of neat plugins:
- https://datasette.io/plugins/datasette-geojson-map shows the geometry from SpatiaLite columns on a map
- https://datasette.io/plugins/datasette-leaflet-geojson can be used to display inline maps next to each column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,
https://github.com/simonw/datasette/issues/339#issuecomment-1074479932,https://api.github.com/repos/simonw/datasette/issues/339,1074479932,IC_kwDOBm6k_c5AC0M8,9599,2022-03-21T22:22:34Z,2022-03-21T22:22:34Z,OWNER,Closing this as obsolete since Datasette no longer uses Sanic.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",340396247,