issues

Exceptions are normal expected behavior when typecasting an invalid format. However, r.parsedate() is really just re-formatting strings and keeping the type as text. So it may be better to print-and-pass on exception so the user can see a complete list of invalid values -- while also allowing for the parser to reformat the remaining valid values. sqlite-utils convert idfpr.db license "Expiration Date" "r.parsedate(value)" [#######-----------------------------] 21% 00:01:57Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/db.py", line 2336, in convert_value return fn(v) File "<string>", line 2, in fn File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/recipes.py", line 8, in parsedate parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 1374, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 652, in parse raise ParserError("String does not contain a date: %s", timestr) dateutil.parser._parser.ParserError: String does not contain a date: / / In this case, I had just one variation of an invalid date: ' / / '. But theoretically there could be many values that would have to be fixed one at a time with the current exception handling.

Is it possible to configure the sql_time_limit_ms beyond 60 seconds? It seems queries are still timing out at 60 seconds when sql_time_limit_ms is set to 180000. We have a very large data set and often encounter timeouts when testing new queries from the datasette UI. We are optimizing our database as much as we can, but still may require more than 60 seconds for complex queries.

I'd like to see spatialite geometry columns rendered more sensibly - at the moment they come through as well-known-binary unless you use custom SQL, and WKB isn't of much use to anyone on the web.

In HTML: they should be shown either as simple lat/long (if it's just a point, for example), or as a sensible placeholder if they're more complex geometries.

In JSON: they should be GeoJSON geometries, (which means they can be automatically fed into a leaflet map with no further messing around).

In CSV: they should be WKT.

I briefly wondered if this should go into a plugin, but I suspect it needs hooking in at a deeper level than the plugin architecture will support any time soon.

https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/utils/init.py#L181-L200

```python @contextmanager def sqlite_timelimit(conn, ms): deadline = time.perf_counter() + (ms / 1000) # n is the number of SQLite virtual machine instructions that will be # executed between each check. It's hard to know what to pick here. # After some experimentation, I've decided to go with 1000 by default and # 1 for time limits that are less than 50ms n = 1000 if ms < 50: n = 1

def handler():
    if time.perf_counter() >= deadline:
        return 1

conn.set_progress_handler(handler, n)
try:
    yield
finally:
    conn.set_progress_handler(None, n)

``` How often do I set a time limit of 50 or less? How much slower does it go thanks to this code?

While working on: - https://github.com/simonw/datasette/issues/1657 - https://github.com/simonw/datasette/issues/1439

It became very clear that the least maintainable part of Datasette at the moment is the way routing to the database, table and row views work - in particular the subclassing mechanism with BaseView and DataView, but also the complex variety of ways in which the URL routes capture different named regular expression groups.

Spotted this while working on: - #1677

https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/utils/init.py#L1005-L1021

Follow-on from: - #1675

Refs: - #1660

I'm going to refactor this stuff out and document it so it can be easily used by plugins:

https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176

HI, when dealing with large json file (~170GB) i got the following error Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 1126, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 1051, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 1393, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 752, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/cli.py", line 1300, in memory rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py", line 185, in rows_from_file return rows_from_file(buffered, format=Format.JSON) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py", line 156, in rows_from_file decoded = json.load(fp) File "/usr/lib/python3.9/json/__init__.py", line 293, in load return loads(fp.read(), MemoryError

It's not possible to combine --multi and --dry-run flag in the convert command.

Let's first create a simple database from JSON string

console $ echo '[{"foo": "abc"}]' | sqlite-utils insert demo.db demo - $ sqlite-utils query demo.db "SELECT * FROM demo" [{"foo": "abc"}]

and then try to convert the "foo" column with a static value "bar" (see docs Converting a column into multiple columns)

console $ sqlite-utils convert demo.db demo foo '{"foo": "bar"}' --multi --dry-run Traceback (most recent call last): File "/home/dotcs/anaconda3/envs/tools/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/sqlite_utils/cli.py", line 2686, in convert for row in db.conn.execute(sql, where_args).fetchall(): sqlite3.OperationalError: user-defined function raised exception

But without the --dry-run flag it does work as expected:

console $ sqlite-utils convert demo.db demo foo '{"foo": "bar"}' --multi $ sqlite-utils query demo.db "SELECT * FROM demo" [{"foo": "bar"}]

console $ sqlite-utils --version sqlite-utils, version 3.25.1