Thanks for opening an issue - this should help future Google searchers figure out what's going on here.

Another approach here could be to store large integers as TEXT in SQLite (or even as BLOB).

Both storing as REAL and storing as TEXT/BLOB feel nasty to me, but it looks like SQLite has a hard upper limit of 9223372036854775807 for integers.

Any progress on this? It would be very helpful on my end as well. Thanks!

And a GIF of the fix after applying: - #598

Datasette is using that now, see: - #1893

@simonw Any chance we can get this tiny patch merged for an upcoming release?

The other thing that could work is something like this: bash export AUTH_TOKENS_DB="tokens" datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB So -e is an alternative version of -s which reads from the named environment variable instead of having the value provided directly as the second value in the pair.

I quite like this, because it could replace the really ugly $ENV pattern we have in plugin configuration at the moment: https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values yaml plugins: datasette-auth-github: client_secret: $env: GITHUB_CLIENT_SECRET

I'm happy to debug and land a patch if it's welcome.

Yes please! What an odd bug.

@dependabot rebase

It's going to be called datasette.json and the concept of metadata will be split out separately. See:

• 2149

@dependabot recreate

Suggestion from @asg017 is that we say that if your row has a null primary key you don't get a link to a row page for that row.

Which has some precedent, because our SQL view display doesn't link to row pages at all (since they don't make sense for views): https://latest.datasette.io/fixtures/simple_view

This also makes it simple to separate out secrets.

datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml

settings.yaml settings: default_page_size: 10 max_returned_rows: 3000 sql_time_limit_ms": 8000 plugins: datasette-ripgrep: path: /usr/local/lib/python3.11/site-packages

secrets.yaml plugins: datasette-auth-github: client_secret: SUCH_SECRET

db-docs.yaml databases: docs: permissions: create-table: id: editor

db-fixtures.yaml databases: fixtures: tables: no_primary_key: hidden: true queries: neighborhood_search: sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood; title: Search neighborhoods description_html: |- <p>This demonstrates <em>basic</em> LIKE search

What version of Datasette are you running?

That feature was added in Datasette 1.0a2, so if you're on the current stable release you won't have it yet.

+1 to getting this merged down.

For future googlers, I installed by... git clone git@github.com:RhetTbull/dogsheep-photos.git cd dogsheep-photos git checkout update_for_bigsur python setup.py install

Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

I just did a github search for user:simonw "def extra_js_urls(" ! Though I'm sure other plugins made by people other than Simon also exist out there https://github.com/search?q=user%3Asimonw+%22def+extra_js_urls%28%22&type=code

I agree with not liking metadata.json stuff in a datasette.* config file. Editing description of a table/column in a file like datasette.* seems odd to me.

Though since plugin configuration currently lives in metadata.json, I think it should be removed from there and placed in datasette.*, at least for top-level config like datasette-auth-github's config. Keeping metadata.json strictly for documentation/licensing/column units makes sense to me, but anything plugin related should be in some config file, like datasette.*.

And ya, supporting both datasette.* and CLI flags makes a lot of sense to me. Any --setting flag should override anything in datasette.* for easier debugging, with possibly a warning message so people don't get confused. Same with --port and a port defined in datasette.*

@dependabot recreate

Using this thread and some other resources I managed to cobble together a couple of sqlite-utils lines to add a geometry column for a table that already has a lat/lng column.

```

add a geometry column

sqlite-utils add-geometry-column [db name] [table name] geometry --type POINT --srid 4326

add a point for each row to geometry column

sqlite-utils --load-extension=spatialite [db name] 'update [table name] SET Geometry=MakePoint(longitude, latitude, 4326);' ```

Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?

The latter - that you could return a promise of arrays, so it parallels the "await me maybe" pattern in Datasette, where you can return either a value, a callable or an awaitable.

I have a hunch that what you're describing might be achievable without adding Promises to the API with something

Oops, I did a poor job explaining. Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

This isn't that big of a deal - it'd be a nice ergonomic improvement, but nowhere near as a big of an improvement as having an officially sanctioned way to add stuff to the column menus in the first place.

This could also be layered on in a future commit without breaking v1 users, too, so it's not at all urgent.

especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag

Ah, this is maybe the the key point. Since it's all hosted inside Datasette, Datasette can provide some arbitrary sugar to make it easier to work with.

My experience with async scripts in JS is that people sometimes don't understand the race conditions inherent to them. If they copy/paste from a tutorial, it does just work. But then they'll delete half the code, and by chance it still works on their machine/Datasette templates, and now someone's headed for an annoying debugging session -- maybe them, maybe someone else who tries to re-use their plugin.

Again, a fairly minor thing, though.

That original example passes against main now.

Seeing as sqlite-utils doesn't currently provide mechanisms for adding check constraints like this I'm going to leave this - I'm happy with the fix I put in for the not null constraints.

Thanks!

Wow, this is a neat feature I didn't know about. Looks like there are a bunch of options:

• NO ACTION (default)
• RESTRICT: application is prohibited from deleting a parent key when there exists one or more child keys mapped to it
• SET NULL: when a parent key is deleted the child key columns of all rows in the child table that mapped to the parent key are set to contain SQL NULL values
• SET DEFAULT: set a specific default
• CASCADE: propagates the delete or update operation on the parent key to each dependent child key

@dependabot recreate

I think I see what is happening here, although I haven't quite work out a fix yet. Usually:

• click.progressbar.render_progress() renders the cursor invisible on each invocation (update of the bar)
• When the progress bar goes out of scope, the __exit()__ method is invoked, which calls render_finish() to make the cursor re-appear.

(See terminal escape sequences BEFORE_BAR and AFTER_BAR in click).

However the sqlite-utils utils.file_progress context manager wraps click.progressbar and yields an instance of a helper class:

python @contextlib.contextmanager def file_progress(file, silent=False, **kwargs): ... with click.progressbar(length=file_length, **kwargs) as bar: yield UpdateWrapper(file, bar.update)

The yielded UpdateWrapper goes out of scope quickly and click.progressbar.__exit__() is called. The cursor is made un-invisible. Hoewever bar is still live and so when the caller iterates on the yielded wrapper this invokes the bar's update method, calling render_progress(), each time printing the "make cursor invisible" escape code. The progressbar.__exit__ function is not called again, so the cursor doesn't re-appear.

This is pretty annoying for FTS because sqlite throws an error instead of just doing something like returning all or no results. This makes users who are unfamiliar with SQL and Datasette think the canned query page is broken and is a frequent source of confusion.

To anyone dealing with this: My solution is to modify the canned query so that it returns no results which cues people to fill in the blank parameters.

So instead of emails_fts match escape_fts(:search))

My canned queries now look like this:

emails_fts match escape_fts(iif(:search=="", "*", :search))

There are no asterisks in my data so the result is always blank.

Ultimately it would be nice to be able to handle this in the metadata. Either making some named parameters required or setting some default values.

I think this feature was removed in Datasette 0.61 and moved to a plugin. People who want hashed URLs can use the datasette-hashed-urls plugin to achieve the same affect.

It looks like you're trying to disable hashed urls, so I think you can just remove that config setting and things will work.

I think it's not currently possible: sqlite-utils requires that it be one of integer, text, float, blob (see code)

IMO, this is a bit of friction and it would be nice if it was more permissive. SQLite permits developers to use any data type when creating a table. For example, this is a perfectly cromulent sqlite session that creates a table with columns of type baz and bar:

``` sqlite> create table foo(column1 baz, column2 bar); sqlite> .schema foo CREATE TABLE foo(column1 baz, column2 bar); sqlite> select * from pragma_table_info('foo'); cid name type notnull dflt_value pk

0 column1 baz 0 0
1 column2 bar 0 0
```

The idea is that the application developer will know what meaning to ascribe to those types. For example, I'm working on a plugin to Datasette. Dates are tricky to handle. If you have some existing rows, you can look at the values in them to know how a user is serializing the dates -- as an ISO 8601 string? An RFC 3339 string? With millisecond precision? With timezone offset? But if you don't yet have any rows, you have to guess. If the column is of type TEXT, you don't even know that it's meant to hold a date! In this case, my plugin will look to see if the column is of type DATE or DATETIME, and assume a certain representation when writing.

Perhaps there is an argument that sqlite-utils is trying to conform to SQLite's strict mode, and that is why it limits the choices. In strict mode, SQLite requires that the data type be one of INT, INTEGER, REAL, TEXT, BLOB, ANY. But that can't be the case -- sqlite-utils supports FLOAT, which is not one of the valid types in strict mode, and it rejects INT, REAL and ANY, which are valid.

i'll adopt this PR to make the changes @simonw suggested https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932

Aha, it's user error on my part.

Adding

sqlite3_db_config.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_int, ctypes.c_int]

makes it work reliably both on the CLI and from datasette, and now I can reproduce the errors you mentioned in the issue description.

Hey @lovasoa, I hope you don't mind - I pulled this PR into datasette-ui-extras, a plugin I'm making that collects UI tweaks to Datasette.

You can apply it to your own Datasette instance by running datasette install datasette-ui-extras

Idea for supporting streaming with the register_output_renderer hook:

python @hookimpl def register_output_renderer(datasette): return { "extension": "test", "render": render_demo, "can_render": can_render_demo, "render_stream": render_demo_stream, # This is new } So there's a new "render_stream" key which can be returned, which if present means that the output renderer supports streaming.

I'll play around with the design of that function signature in:

• 1999

• 1062

A bit late to this, but I have made an app to publish air quality data in Bristol, UK. air quality data in Bristol, UK. Next step to see if I can make a streamlit app based on this to produce some nice charts.

When you run --root you need to follow the special link that gets output to the console:

% datasette --root http://127.0.0.1:8001/-/auth-token?token=036d8055cc8000e9667f21c1dd08722a9358c066463873ad9566d23d88765c52 INFO: Started server process [53934] INFO: Waiting for application startup. INFO: Application startup complete. That /-/auth-token?... link is the one that sets the cookie and lets you in.

I guess it is not incorrect when it says the version is 4, though it is confusing. Maybe it doesn't even refer to FTS4/FTS5 versions, but something else? In any case, it's not related to sqlite-utils, but SQLite itself.

Oh this is with datasette package? That should work. Will investigate.

Alright, added Cmd+Enter to submit (Ctrl+Enter on Windows as well bc of using Meta-Enter on codemirror). We can make that MacOS only by changing the combo to Cmd+Enter specifically but I think it's probably fine to have both.

The main issue is that we don't pass the relevant table data down to QueryView.

If you can come up with a static example JSON data structure example that does the right thing, I'm happy to refactor QueryView to make that available to the template - or even have a separate fetch() that grabs just the data needed for the autocomplete as a separate hit when the page loads (whichever has better performance implications). I'm working a fair amount in the view classes at the moment so adding this to that work would make sense.

Actually no, I'm going to add a class of details-menu to the other details elements that SHOULD be closed. That way custom templates using <details> won't close in a surprising way.

The _internal database is intended to help Datasette handle much larger attached databases. Right now Datasette attempts to show every database on the https://latest.datasette.io/ index page and every table on the https://latest.datasette.io/fixtures database index page - but these are not paginated. If you had a database containing 1,000 tables the database index page would get pretty slow.

So I want to be able to paginate (and search) those. But to paginate them it's useful to have them in a database table itself, since then I can paginate using SQL.

My plan for _internal is to use it to implement those advanced browsing features. I've not completed this work yet though. See this issue for more details on that:

• 417

I think you may have misunderstood this feature. This is talking about the _internal in-memory database, which maintains a set of tables that list the databases and tables that are attached to Datasette.

They're not a copy of the data itself - just a list of table names, column names and database names.

You can see what that database looks like by signing in as root - running datasette --root and clicking the link. Or you can see an example here:

• Click the button on https://latest.datasette.io/login-as-root
• Now visit https://latest.datasette.io/_internal

For the example instance that looks like this:

The two most interesting tables in there are these ones:

As you can see, it's just the table schema itself and the columns that make up the tables. Even if you have hundreds of databases connected each with hundreds of tables this should still only add up to a few MB of RAM.

One idea: add a /-/debug page (or /-/tips or /-/checks) which shows the incoming requests headers and could even detect if there's an x-forwarded-host header that isn't being repeated and show a tip on how to fix that.

@dependabot rebase

I think mine has a better pattern for handling /* ... anything in here that isn't */ ... */

Yeah we should fix this.

https://www.sqlite.org/lang_comment.html - SQLite also supports -- style comments.

I like how explicit the documentation is here:

SQL comments begin with two consecutive "-" characters (ASCII 0x2d) and extend up to and including the next newline character (ASCII 0x0a) or until the end of input, whichever comes first.

C-style comments begin with "/" and extend up to and including the next "/" character pair or until the end of input, whichever comes first. C-style comments can span multiple lines.

The argument that has always convinced me NOT to use target="_blank" (even for links like this one) is that it breaks browser expectations.

If you click a link with `target="_blank" on it you get a new browser window... with a disabled back button. You have to then know to close that browser window in order to return to the previous page - as opposed to hitting the "back" button like usual.

You'll note that Datasette doesn't use target="_blank" even on URLs presented in database tables - like these ones: https://latest.datasette.io/fixtures/roadside_attractions

So I'm very firmly in the anti-target-blank camp!

This is the kind of change which I'd suggest implementing as a plugin. datasette-external-links-new-windows could run a bit of JavaScript on every page that looks for <a href="..."> elements that link to off-domain pages and adds target="_blank" to them via the DOM.

That way people who like target="_blank" can have it!

It might have been useful for Datasette to show an error when started against a settings.json file that contains an invalid setting though.

Should consider how this could best handle creating columns that are integer and float as opposed to just text.

https://discord.com/channels/823971286308356157/823971286941302908/1019630014544748584 is a relevant discussion on Discord. Even if you create the schema in advance with the correct column types, this import mechanism can put empty strings in blank float/integer columns when ideally you would want to have nulls.

Related feature idea for sqlite-utils transform: - #488

Not sure how best to handle this for sqlite3 .import imports.

OK, I'm convinced that it's time to start figuring this out.

I've done a little bit of this with Django in the past, but Datasette isn't built on Django.

It looks to me like the key library for implementing this is Babel: https://babel.pocoo.org/en/latest/

It's been around since 2007 and is very widely used: https://github.com/python-babel/babel/network/dependents?package_id=UGFja2FnZS01MDM0NTU3NQ%3D%3D

Also found these hints on getting it to work with Jinja: https://stackoverflow.com/questions/12046998/babel-doesnt-recognize-jinja2-extraction-method-for-language-support

That was clearly the intention from the description of this issue: - #4

Thanks!

Tested that with:

datasette publish cloudrun fixtures.db --service issue-1779 --min-instances 2 --max-instances 4

@dependabot rebase

I should warn you that this isn't a supported API - I reserve the right to change how it works between release without a major version bump, because it's not part of the documented API surface.

You'll be fine if you pin to exact versions of the library though!

You may find this recently-documented function useful though: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file

See: - #443

I'm going to close this issue for the moment, but if anyone wants to submit a PR that cleans up this I'll happily review it.

Relevant TIL: https://til.simonwillison.net/sqlite/one-line-csv-operations

I don't think this change needs any changes to the documentation: https://docs.datasette.io/en/stable/custom_templates.html#custom-templates

Yeah, at the moment the best way to do this is with search_sql(), but you're right it really isn't very intuitive.

Here's how I would do this, using a CTE trick to combine the queries: python search_sql = db["articles"].search_sql(columns=["title", "author"])) sql = f""" with search_results as ({search_sql}) select * from search_results where owner = :owner """ results = db.query(sql, {"query": "my search query", "owner": "my owner"}) I'm not sure if sqlite-utils should ever evolve to provide a better way of doing this kind of thing to be honest - if it did, it would turn into more of an ORM. Something like PeeWee may be a better option here.

Updated docs: https://docs.datasette.io/en/latest/getting_started.html#using-datasette-on-your-own-computer and https://docs.datasette.io/en/latest/cli-reference.html#datasette-serve-help

Some of the things I'd like to use ?_extra= for, that may or not make sense as plugins:

• Performance breakdown information, maybe including explain output for a query/table
• Information about the tables that were consulted in a query - imagine pulling in additional table metadata
• Statistical aggregates against the full set of results. This may well be a Datasette core feature at some point in the future, but being able to provide it early as a plugin would be really cool.
• For tables, what are the other tables they can join against?
• Suggested facets
• Facet results themselves
• New custom facets I haven't thought of - though the register_facet_classes hook covers that already
• Table schema
• Table metadata
• Analytics - how many times has this table been queried? Would be a plugin thing
• For geospatial data, how about a GeoJSON polygon that represents the bounding box for all returned results? Effectively this is an extra aggregation.

Looking at https://github-to-sqlite.dogsheep.net/github/commits.json?_labels=on&_shape=objects for inspiration.

I think there's a separate potential mechanism in the future that lets you add custom columns to a table. This would affect .csv and the HTML presentation too, which makes it a different concept from the ?_extra= hook that affects the JSON export (and the context that is fed to the HTML templates).

That's interesting - yeah it looks like the number of pages can be derived from the Link header, which is enough information to show a progress bar, probably using Click just to avoid adding another dependency.

https://docs.github.com/en/rest/guides/traversing-with-pagination

I'm going to close this ticket since it looks like this is a bug in the way the Dockerfile builds Python, but I'm going to ship a fix for that issue I found so the LD_PRELOAD workaround above should work OK with the next release of sqlite-utils. Thanks for the detailed bug report!

I like your design, though I think it should be "nomodule": True for consistency with the other options.

I think "async": True is worth supporting too.

Yup, you're right in what you figured out here: stand-alone plugins can't currently package static assets other then using the static folder.

The datasette-plugin cookiecutter template should make creating a Python package pretty easy though: https://github.com/simonw/datasette-plugin

You can run that yourself, or you can run it using this GitHub template repository: https://github.com/simonw/datasette-plugin-template-repository

I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end.

Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :)

All good. Report back any issues you find with this stuff. Metadata/dynamic config hasn't been tested widely outside of what I've done AFAIK. If you find a strong use case for async meta, it's going to be better to know sooner rather than later!

I would love this merged!

Hi @simonw,

I've received some inquiries over the last year or so about Datasette and how it might be supported by Mangum. I maintain Mangum which is, as far as I know, the only project that provides support for ASGI applications in AWS Lambda.

If there is anything that I can help with here, please let me know because I think what Datasette provides to the community (even beyond OSS) is noble and worthy of special consideration.

I like the idea of having stock conversions you could import. I'd actually move them to a dedicated module (call it sqlite_utils.conversions or something), because it's different from other utilities. Maybe they even take configuration, or they're composable.

```python from sqlite_utils.conversions import LongitudeLatitude

db["places"].insert( { "name": "London", "lng": -0.118092, "lat": 51.509865, }, conversions={"point": LongitudeLatitude("lng", "lat")}, ) ```

I would definitely use that for every CSV I get with lat/lng columns where I actually need GeoJSON.

from sqlite_utils.utils import find_spatialite is part of the documented API already:

https://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite

To avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a from .gis import find_spatialite line at the top of utils.py such that any existing code with that documented import continues to work.

The existing flow where you can apply filters to a table and then click "View and edit SQL" to see the query is a good starting point.

Group by queries are both crucially important and difficult to assemble for beginners. Providing a way to see the query that was used by a facet (since facets are really just group-by-counts) would be very useful, which could come out of this:

• 1080

https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for "You are looking at v0.36 but you should be looking at 0.40" but doesn't hand the case I need here which is "you are looking at /latest/ but you should be looking at /stable/".

Correction! That tool DOES support that, as can be seen in their example configuration for their own documentation:

https://github.com/humitos/sphinx-version-warning/blob/a82156c2ea08e5feab406514d0ccd9d48a345f48/docs/conf.py#L32-L38

python versionwarning_messages = { 'latest': 'This is a custom message only for version "latest" of this documentation.', } versionwarning_admonition_type = 'tip' versionwarning_banner_title = 'Tip' versionwarning_body_selector = 'div[itemprop="articleBody"]'

This is definitely a missing feature. The "different types of facet" stuff feels incomplete to me generally - this is one issue, but this one as well:

• 625

Good call! I'm doing a refactor #1518 right now which will hopefully bring the functionality of those two much closer - I'll make a note to consider this there too.

One option: allow CODE to be a special value of - which means "read from standard input". It's a tiny bit of a hack but I think it would work here.

If you wanted to replace a column entirely with hyphens you would still be able to do this:

sqlite-utils convert my.db mytable col1 '"-"'

I actually think it would be a useful thing to add support for in datasette. It wouldn't be difficult to unwind an array of params and add the placeholders automatically.

There isn't any way to do this with sqlite as far as I know. The only option is to insert the right number of ? placeholders into the sql template and then provide an array of values.

I tried deploying the most recent version of the Dockerfile in this thread (link to comment), and after trying a few different different combinations, I was only successful when I used --no-cpu-throttling ("CPU Is always allocated" in the UI)

Using this method, I got a very similar issue to you: The first time I'd load the site I'd get a 503. But after that first load, I didn't get the issue again. It would re-occur if the service started from cold boot.

I suspect this is a race condition in the supervisord configuration. The errors I got were the same Connection refused: AH00957: http: attempt to connect to 127.0.0.1:8001 (127.0.0.1) failed, and that seems to indicate that datasette hadn't yet started.

Looking at the order of logs getting back, the processes reported successfully completing loading after the first 503 was returned, so that makes me think race condition.

I can replicate this locally, if I docker run and request localhost:5000/prefix before I get the datasette entered RUNNING state message. Cloud Run wakes up when requests are received, so this test would semi-replicate that, but local docker would be the equivalent of a persistent process, hence it doesn't normally exhibit the same issues.

Unfortunately supervisor/supervisor issue 122 (not linking as to prevent cross-project link spam) seems to say that dependency chaining is a feature that's been asked for for a long time, but hasn't been implemented. You could try some suggestions in that thread.

I've never used DEFAULT 'CURRENT_TIMESTAMP' myself so this one should be an interesting bug to explore.

i'm going to take a swing at this today. we'll see.

@dependabot recreate

Thanks for this!

Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-directly-against-csv-or-json

If two files have the same name they will be assigned a numeric suffix:

$ sqlite-utils memory foo/data.csv bar/data.csv "select * from data_2"

The bug is in this code: https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/sqlite_utils/cli.py#L2205-L2227

I'll add researchers too.

I love it! maybe 'researchers' instead? Or 'scientists and researchers'?

I'm going with "columns": {"name-of-column": "description-of-column"}.

If I decide to make "col" and "nocol" available in metadata I'll use those as the keys in the metadata, for consistency with the existing query string parameters.

I'm OK with having both "columns": ... and "col": ... keys in the metadata, even though they could be a tiny bit confusing without the documentation.

This is a feature already! You can do this:

sqlite-utils insert nl-demo.db mytable data.ndjson --nl

See https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-newline-delimited-json

I think I made a mistake in this commit: https://github.com/simonw/datasette/commit/0486303b60ce2784fd2e2ecdbecf304b7d6e6659

It looks like I copied $VERSION_TAG from here - but it's not available in the publish.yml flow: https://github.com/simonw/datasette/blob/0486303b60ce2784fd2e2ecdbecf304b7d6e6659/.github/workflows/push_docker_tag.yml#L18-L25

A workaround could be to define a bash or zsh alias of some sort.

Solution: sqlite-utils memory - attempts to detect the input based on if it starts with a { or [ (likely JSON) or if it doesn't use the csv.Sniffer() mechanism. Or you can use sqlite-utils memory -:csv to specifically indicate the type of input.

There is a feature for this at the moment, but it's a little bit hidden: you can use ?_json=col to tell Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments

Thanks :)

I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning string instead of the expected dictionary and breaking your code.

If a developer is not sure if the JSON fields are valid, but then retrieves and parse them, it should handle errors too. Handling inconsistent data is necessary due to the nature of SQLite. A global or dataset option to render the data as they have been defined (JSON, boolean, etc.) when requesting JSON could allow the user to download a regular JSON from the browser without having to rely on APIs. I would guess someone could just make a custom template with an extra JSON-parsed download button otherwise :)

There is a feature for this at the moment, but it's a little bit hidden: you can use ?_json=col to tell Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments

I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning string instead of the expected dictionary and breaking your code.

I think this is more likely to happen in Datasette than in sqlite-utils - see https://github.com/simonw/datasette/issues/1356 for thoughts on this.

Another potential way to implement this would be to hold the SQLite connection open and execute the full query there.

I've avoided this in the past due to concerns of resource exhaustion - if multiple requests attempt this at the same time all of the connections in the pool will become tied up and the site will be unable to respond to further requests.

But... now that Datasette has authentication there's the possibility of making this feature only available to specific authenticated users - the --root user for example. Which avoids the danger while unlocking a super-useful feature.

Not to mention people who are running Datasette privately on their own laptop, or the proposed --query CLI feature in #1356.

@dependabot recreate

Fixed: https://simonwillison.net/2020/Sep/23/sqlite-advanced-alter-table/

I'm going to merge this and add a test - thanks!

This worked for me:
<td class="col-{{ cell.column|to_css_class }} type-{{ cell.value_type }}">{{ cell.value | replace('", "','; ') | replace('[\"','') | replace('\"]','')}}</td>

I'm sure there is a prettier (and more flexible) way, but for now, this is ever-so-much more pleasant to look at.

------ AFTER:

------ BEFORE:

(Note: I didn't figure out how to have one item have no semicolon, while multi-items close with a semicolon, but this is good enough for now. I also didn't figure out how to set up a new jinja filter. I don't want to add to /datasette/utils/init.py as I assume that would get overwritten when upgrading datasette. Having a starter guide on creating jinja filters in datasette would be helpful. (The jinja documentation isn't datasette-specific enough for me to quite nail it.)

Hello again,

I was able to get my facets running with this settings.json, which was lifted from one of Simon's datasette's and slightly modified.

{ "default_page_size": 100, "max_returned_rows": 1000, "num_sql_threads": 3, "sql_time_limit_ms": 9000, "default_facet_size": 10, "facet_time_limit_ms": 9000, "facet_suggest_time_limit_ms": 500, "hash_urls": false, "allow_facet": true, "suggest_facets": false, "default_cache_ttl": 5, "default_cache_ttl_hashed": 31536000, "cache_size_kb": 0, "allow_csv_stream": true, "max_csv_mb": 100, "truncate_cells_html": 2048, "template_debug": false, "base_url": "/" }

Idea: default to displaying single-dimension JSON arrays of strings as a comma-separated list but show the comma in a different colour - something like this:

I used this HTML for the prototype (re-using .type-int just to get the colour): ```html

```