issue_comments

Closing this as obsolete since Datasette no longer uses Sanic.

I'm closing this issue because this is now solved by a number of neat plugins:

• https://datasette.io/plugins/datasette-geojson-map shows the geometry from SpatiaLite columns on a map
• https://datasette.io/plugins/datasette-leaflet-geojson can be used to display inline maps next to each column

Thinking about options for fixing this...

The following query works fine: sql select * from test_view where cast(has_expired as text) = '1' I don't want to start using this for every query, because one of the goals of Datasette is to help people who are learning SQL: - #1613

If someone clicks on "View and edit SQL" from a filtered table page I don't want them to have to wonder why that cast is there.

But... for querying views, the cast turns out to be necessary.

So one fix would be to get the SQL generating logic to use casts like this any time it is operating against a view.

An even better fix would be to detect which columns in a view come from a table and which ones might not, and only use casts for the columns that aren't definitely from a table.

The trick I was exploring here might be able to help with that: - #1293

I wonder if this will be a problem with generated columns, or with SQLite strict tables?

My hunch is that strict tables will continue to work without any changes, because https://www.sqlite.org/stricttables.html says nothing about their impact on comparison operations. I should test this to make absolutely sure though.

Generated columns have a type, so my hunch is they will continue to work fine too.

Relevant section of the SQLite documentation: 3.2. Affinity Of Expressions:

When an expression is a simple reference to a column of a real table (not a VIEW or subquery) then the expression has the same affinity as the table column.

In your example, has_expired is no longer a simple reference to a column of a real table, hence the bug.

Then 4.2. Type Conversions Prior To Comparison fills in the rest:

SQLite may attempt to convert values between the storage classes INTEGER, REAL, and/or TEXT before performing a comparison. Whether or not any conversions are attempted before the comparison takes place depends on the type affinity of the operands.

Oh this is fascinating! I replicated the bug (thanks for the steps to reproduce) and it looks like this is down to the following:

Against views, where has_expired = 1 returns different results from where has_expired = '1'

This doesn't happen against tables because of SQLite's type affinity mechanism, which handles the type conversion automatically.

I'm going to change the original logic to set n=1 for times that are <= 20ms - and update the comments to make it more obvious what is happening.

Oh interesting, it turns out there is ONE place in the code that sets the ms to less than 20 - this test fixture: https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/tests/fixtures.py#L224-L226

Here's another microbenchmark that measures how many nanoseconds it takes to run 1,000 vmops:

```python import sqlite3 import time

db = sqlite3.connect(":memory:")

i = 0 out = []

def count(): global i i += 1000 out.append(((i, time.perf_counter_ns())))

db.set_progress_handler(count, 1000)

print("Start:", time.perf_counter_ns()) all = db.execute(""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """).fetchall() print("End:", time.perf_counter_ns())

print() print("So how long does it take to execute 1000 ops?")

prev_time_ns = None for i, time_ns in out: if prev_time_ns is not None: print(time_ns - prev_time_ns, "ns") prev_time_ns = time_ns Running it: % python nanobench.py Start: 330877620374821 End: 330877632515822

So how long does it take to execute 1000 ops? 47290 ns 49573 ns 48226 ns 45674 ns 53238 ns 47313 ns 52346 ns 48689 ns 47092 ns 87596 ns 69999 ns 52522 ns 52809 ns 53259 ns 52478 ns 53478 ns 65812 ns ``` 87596ns is 0.087596ms - so even a measure rate of every 1000 ops is easily finely grained enough to capture differences of less than 0.1ms.

If anything I could bump that default 1000 up - and I can definitely eliminate the if ms < 50 branch entirely.

OK here's a microbenchmark script: ```python import sqlite3 import timeit

db = sqlite3.connect(":memory:") db_with_progress_handler_1 = sqlite3.connect(":memory:") db_with_progress_handler_1000 = sqlite3.connect(":memory:")

db_with_progress_handler_1.set_progress_handler(lambda: None, 1) db_with_progress_handler_1000.set_progress_handler(lambda: None, 1000)

def execute_query(db): cursor = db.execute(""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """) list(cursor.fetchall())

print("Without progress_handler") print(timeit.timeit(lambda: execute_query(db), number=100))

print("progress_handler every 1000 ops") print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1000), number=100))

print("progress_handler every 1 op") print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1), number=100)) Results: % python3 bench.py Without progress_handler 0.8789225700311363 progress_handler every 1000 ops 0.8829826560104266 progress_handler every 1 op 2.8892734259716235 ```

So running every 1000 ops makes almost no difference at all, but running every single op is a 3.2x performance degradation.

David Raymond solved it there: https://sqlite.org/forum/forumpost/330c8532d8a88bcd

Don't forget to step through the results. All .execute() has done is prepared it.

db.execute(query).fetchall()

Sure enough, adding that gets the VM steps number up to 190,007 which is close enough that I'm happy.

Maybe there is a better name for this method that helps emphasize its cascading nature.

Posed a question about that here: https://sqlite.org/forum/forumpost/de9ff10fa7

Here's the Python-C implementation of set_progress_handler: https://github.com/python/cpython/blob/4674fd4e938eb4a29ccd5b12c15455bd2a41c335/Modules/_sqlite/connection.c#L1177-L1201

It calls sqlite3_progress_handler(self->db, n, progress_callback, ctx);

https://www.sqlite.org/c3ref/progress_handler.html says:

The parameter N is the approximate number of virtual machine instructions that are evaluated between successive invocations of the callback X

So maybe VM-steps and virtual machine instructions are different things?

This is weird: ```python import sqlite3

db = sqlite3.connect(":memory:")

i = 0

def count(): global i i += 1

db.set_progress_handler(count, 1)

db.execute(""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """)

print(i) Outputs `24`. But if you try the same thing in the SQLite console: sqlite> .stats vmstep sqlite> with recursive counter(x) as ( ...> select 0 ...> union ...> select x + 1 from counter ...> ) ...> select * from counter limit 10000; ... VM-steps: 200007 ```

How long does it take for SQLite to execute 1000 opcodes anyway?

So it looks like even for facet suggestion n=1000 always - it's never reduced to n=1.

https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/app.py#L118-L122 sets it to 50ms for facet suggestion but that's not going to pass ms < 50:

python Setting( "facet_suggest_time_limit_ms", 50, "Time limit for calculating a suggested facet", ),

I've simplified this a ton now. I'm going to keep working on this in the long-term but I think this issue can be closed.

Documentation: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-resource-none

BaseView is looking a LOT slimmer now that I've moved all of the permissions stuff out of it.

That makes sense; just a little hint that points folks towards doing the right thing might be helpful!

fwiw, the reason I was using jq in the first place was just a quick way to extract one attribute from an actual JSON array. When I initially imported it, I got a table with a bunch of embedded JSON values, rather than a native table, because each array entry had two attributes, one with the data I actually wanted. Not sure how common a use-case this is, though (and easily fixed, aside from the jq weirdness!)

I've not really thought about standards as much here as I should. It looks like there are two competing specs for newline-delimited JSON!

http://ndjson.org/ is the one I've been using in sqlite-utils - and https://github.com/ndjson/ndjson-spec#31-serialization says:

The JSON texts MUST NOT contain newlines or carriage returns.

https://jsonlines.org/ is the other one. It is slightly less clear, but it does say this:

1. Each Line is a Valid JSON Value

The most common values will be objects or arrays, but any JSON value is permitted.

My interpretation of both of these is that newlines in the middle of a JSON object shouldn't be allowed.

So what's jq doing here? It looks to me like that jq format is its own thing - it's not actually compatible with either of those two loose specs described above.

The jq docs seem to call this "whitespace-separated JSON": https://stedolan.github.io/jq/manual/v1.6/#Invokingjq

The thing I like about newline-delimited JSON is that it's really trivial to parse - loop through each line, run it through json.loads() and that's it. No need to try and unwrap JSON objects that might span multiple lines.

Unless someone has written a robust Python implementation of a jq-compatible whitespace-separated JSON parser, I'm inclined to leave this as is. I'd be fine adding some documentation that helps point people towards jq -c though.

https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/views/base.py#L69-L77

This is weirdly different from how check_permissions() used to work, in that it doesn't differentiate between None and False.

https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L79-L103

When looking at this code earlier I assumed that the following would check each permission in turn and fail if any of them failed: python await self.ds.ensure_permissions( request.actor, [ ("view-table", (database, table)), ("view-database", database), "view-instance", ] ) But it's not quite that simple: if any of them fail, it fails... but if an earlier one returns True the whole stack passes even if there would have been a failure later on!

If that is indeed the right abstraction, I need to work to make the documentation as clear as possible.

This method here: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/app.py#L632-L664

Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions

This method allows multiple permissions to be checked at onced. It raises a datasette.Forbidden exception if any of the checks are denied before one of them is explicitly granted.

This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns True or not a single one of them returns False:

That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.

Also calling that function permissions_allowed() is confusing because there is a plugin hook with a similar name already: https://docs.datasette.io/en/stable/plugin_hooks.html#permission-allowed-datasette-actor-action-resource

Idea: ds.permission_allowed() continues to just return True or False.

A new ds.ensure_permissions(...) method is added which raises a Forbidden exception if a check fails (hence the different name)`.

One benefit of the current design of check_permissions that raises an exception is that the exception includes information on WHICH of the permission checks failed. Returning just True or False loses that information.

I could return an object which evaluates to False but also carries extra information? Bit weird, I've never seen anything like that in other Python code.

The other difference though is that ds.permission_allowed(...) works against an actor, while check_permission() works against a request (though just to access request.actor).

Though at that point check_permission is such a light wrapper around self.ds.permission_allowed() that there's little point in it existing at all.

So maybe check_permisions() becomes ds.permissions_allowed().

permission_allowed() v.s. permissions_allowed() is a bit of a subtle naming difference, but I think it works.

A slightly odd thing about these methods is that they either fail silently or they raise a Forbidden exception.

Maybe they should instead return True or False and the calling code could decide if it wants to raise the exception? That would make them more usable and a little less surprising.

I'm going to refactor this stuff out and document it so it can be easily used by plugins:

https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103

I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end.

Related: - #1062

I can fix this like so: ``` % sqlite-utils convert demo.db demo foo '{"foo": "bar"}' --multi --dry-run abc --- becomes: {"foo": "bar"}

Would affect 1 row Diff is this:diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 0cf0468..b2a0440 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2676,7 +2676,10 @@ def convert( raise click.ClickException(str(e)) if dry_run: # Pull first 20 values for first column and preview them - db.conn.create_function("preview_transform", 1, lambda v: fn(v) if v else v) + preview = lambda v: fn(v) if v else v + if multi: + preview = lambda v: json.dumps(fn(v), default=repr) if v else v + db.conn.create_function("preview_transform", 1, preview) sql = """ select [{column}] as value, ```

Thanks for the really clear steps to reproduce!

Needs tests and documentation.

Prototype: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 8255b56..0a3693e 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2583,7 +2583,11 @@ def generate_convert_help(): """ ).strip() recipe_names = [ - n for n in dir(recipes) if not n.startswith("") and n not in ("json", "parser") + n + for n in dir(recipes) + if not n.startswith("_") + and n not in ("json", "parser") + and callable(getattr(recipes, n)) ] for name in recipe_names: fn = getattr(recipes, name) diff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py index 6918661..569c30d 100644 --- a/sqlite_utils/recipes.py +++ b/sqlite_utils/recipes.py @@ -1,17 +1,38 @@ from dateutil import parser import json

+IGNORE = object() +SET_NULL = object()

-def parsedate(value, dayfirst=False, yearfirst=False): + +def parsedate(value, dayfirst=False, yearfirst=False, errors=None): "Parse a date and convert it to ISO date format: yyyy-mm-dd" - return ( - parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() - ) + try: + return ( + parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst) + .date() + .isoformat() + ) + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise

-def parsedatetime(value, dayfirst=False, yearfirst=False): +def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None): "Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS" - return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + try: + return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise

def jsonsplit(value, delimiter=",", type=str): ```

This is quite nice: % sqlite-utils convert test-dates.db dates date "r.parsedate(value, errors=r.IGNORE)" [####################################] 100% % sqlite-utils rows test-dates.db dates [{"id": 1, "date": "2016-03-15"}, {"id": 2, "date": "2016-03-16"}, {"id": 3, "date": "2016-03-17"}, {"id": 4, "date": "2016-03-18"}, {"id": 5, "date": "2016-03-19"}, {"id": 6, "date": "2016-03-20"}, {"id": 7, "date": "2016-03-21"}, {"id": 8, "date": "2016-03-22"}, {"id": 9, "date": "2016-03-23"}, {"id": 10, "date": "//"}, {"id": 11, "date": "2016-03-25"}, {"id": 12, "date": "2016-03-26"}, {"id": 13, "date": "2016-03-27"}, {"id": 14, "date": "2016-03-28"}, {"id": 15, "date": "2016-03-29"}, {"id": 16, "date": "2016-03-30"}, {"id": 17, "date": "2016-03-31"}, {"id": 18, "date": "2016-04-01"}] % sqlite-utils convert test-dates.db dates date "r.parsedate(value, errors=r.SET_NULL)" [####################################] 100% % sqlite-utils rows test-dates.db dates [{"id": 1, "date": "2016-03-15"}, {"id": 2, "date": "2016-03-16"}, {"id": 3, "date": "2016-03-17"}, {"id": 4, "date": "2016-03-18"}, {"id": 5, "date": "2016-03-19"}, {"id": 6, "date": "2016-03-20"}, {"id": 7, "date": "2016-03-21"}, {"id": 8, "date": "2016-03-22"}, {"id": 9, "date": "2016-03-23"}, {"id": 10, "date": null}, {"id": 11, "date": "2016-03-25"}, {"id": 12, "date": "2016-03-26"}, {"id": 13, "date": "2016-03-27"}, {"id": 14, "date": "2016-03-28"}, {"id": 15, "date": "2016-03-29"}, {"id": 16, "date": "2016-03-30"}, {"id": 17, "date": "2016-03-31"}, {"id": 18, "date": "2016-04-01"}]

I'm going to try the errors=r.IGNORE option and see what that looks like once implemented.

I think the options here should be:

• On error, raise an exception and revert the transaction (the current default)
• On error, leave the value as-is
• On error, set the value to None

These need to be indicated by parameters to the r.parsedate() function.

Some design options:

• ignore=True to ignore errors - but how does it know if it should leave the value or set it to None? This is similar to other ignore=True parameters elsewhere in the Python API.
• errors="ignore", errors="set-null" - I don't like magic string values very much, but this is similar to Python's str.encode(errors=) mechanism
• errors=r.IGNORE - using constants, which at least avoids magic strings. The other one could be errors=r.SET_NULL
• error=lambda v: None or error=lambda v: v - this is a bit confusing though, introducing another callback that gets to have a go at converting the error if the first callback failed? And what happens if that lambda itself raises an error?

I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction.

Here's the code that does that:

https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L2523-L2526

Then I ran this to convert 2016-03-27 etc to 2016/03/27 so I could see which ones were later converted:

sqlite-utils convert test-dates.db dates date 'value.replace("-", "/")'

Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/ sqlite-utils create-database test-dates.db sqlite-utils create-table test-dates.db dates id integer date text --pk id sqlite-utils test-dates.db "WITH RECURSIVE cnt(x) AS ( SELECT 0 UNION ALL SELECT x+1 FROM cnt LIMIT (SELECT ((julianday('2016-04-01') - julianday('2016-03-15'))) + 1) ) insert into dates (date) select date(julianday('2016-03-15'), '+' || x || ' days') as date FROM cnt;" After running that: % sqlite-utils rows test-dates.db dates [{"id": 1, "date": "2016-03-15"}, {"id": 2, "date": "2016-03-16"}, {"id": 3, "date": "2016-03-17"}, {"id": 4, "date": "2016-03-18"}, {"id": 5, "date": "2016-03-19"}, {"id": 6, "date": "2016-03-20"}, {"id": 7, "date": "2016-03-21"}, {"id": 8, "date": "2016-03-22"}, {"id": 9, "date": "2016-03-23"}, {"id": 10, "date": "2016-03-24"}, {"id": 11, "date": "2016-03-25"}, {"id": 12, "date": "2016-03-26"}, {"id": 13, "date": "2016-03-27"}, {"id": 14, "date": "2016-03-28"}, {"id": 15, "date": "2016-03-29"}, {"id": 16, "date": "2016-03-30"}, {"id": 17, "date": "2016-03-31"}, {"id": 18, "date": "2016-04-01"}] Then to make one of them invalid:

sqlite-utils test-dates.db "update dates set date = '//' where id = 10"