The test there failed because it turns out there's a whole bunch of places that set the Access-Control-Allow-Origin header. I'm going to close this PR and ship a fix that refactors those places to use the same code.

This looks like a good fix to me.

This is a great idea, thanks.

OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting @dependabot ignore this major version or @dependabot ignore this minor version. You can also ignore all major, minor, or patch releases for a dependency by adding an ignore condition with the desired update_types to your config file.

If you change your mind, just re-open this PR and I'll resolve any conflicts on it.

Looks like this PR is closed. If you re-open it I'll rebase it as long as no-one else has edited it (you can use @dependabot reopen if the branch has been deleted).

@dependabot recreate

I think this is the change in httpx which is causing the bug for me: https://github.com/encode/httpx/commit/ff9813e84dab56f0f3c4ef3a159a4cce8c644a91#diff-0d0cbe9ebcd03cc8c780b0407762540a082f70cc64257f2fcd588cc30f43c15cR96

https://user-images.githubusercontent.com/9599/137224339-0aed171f-b351-4d67-8d6c-ad22f2301f18.png">

Previously it was using path from path, _, query = full_path.partition(b"?") to populate the raw_path key - but it changed to instead using request.url.raw_path which presumably implements the logic that includes the query string.

This is weird - as far as I can tell httpx has included the query string in raw_path for well over a year: https://github.com/encode/httpx/commit/8e4a8a1c73f60fe5754f95b308beaa725cb8791d#diff-c9a78eb3b5f5c4fac4e5552165fbdd5320c7e3fadf9eedabcb5461393466c090R235

(Pdb) request.scope['path']
'/_memory.json'
(Pdb) request.scope['raw_path']
b'/_memory.json?sql=select+sqlite_version()'

So raw_path now includes the query string.

I think the issue is in route_path():

> /Users/simon/Dropbox/Development/datasette/datasette/app.py(1182)route_path()
-> response = await view(request, send)
(Pdb) path
'/_memory.json?sql=select+sqlite_version()'

Upgrading to 0.20.0 gives me lots of the following errors:

'{"ok": false, "error": "Database not found: .json?_sort=relationships", "status": 404, "title": null}'

It looks like the full query string is now being treated as the name of the database.

Codecov Report

Merging #330 (fc3de90) into main (718a8f6) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #330   +/-   ##
=======================================
  Coverage   96.59%   96.59%           
=======================================
  Files           5        5           
  Lines        2230     2230           
=======================================
  Hits         2154     2154           
  Misses         76       76           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 718a8f6...fc3de90. Read the comment docs.

Codecov Report

Merging #1463 (2dd94c5) into main (0d5cc20) will decrease coverage by 0.00%.
The diff coverage is n/a.

:exclamation: Current head 2dd94c5 differs from pull request most recent head 481212c. Consider uploading reports for the commit 481212c to get more accurate results

@@            Coverage Diff             @@
##             main    #1463      +/-   ##
==========================================
- Coverage   91.83%   91.83%   -0.01%     
==========================================
  Files          34       34              
  Lines        4422     4421       -1     
==========================================
- Hits         4061     4060       -1     
  Misses        361      361              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e1012e7...481212c. Read the comment docs.

The core problem here is treating the ?_facet= query string parameters as the point of truth for which facets are currently enabled.

Instead, I could use a data- attribute on the displayed facets.

Codecov Report

Merging #1471 (01b05ed) into main (0d5cc20) will decrease coverage by 0.00%.
The diff coverage is n/a.

:exclamation: Current head 01b05ed differs from pull request most recent head 847238a. Consider uploading reports for the commit 847238a to get more accurate results

@@            Coverage Diff             @@
##             main    #1471      +/-   ##
==========================================
- Coverage   91.83%   91.83%   -0.01%     
==========================================
  Files          34       34              
  Lines        4422     4421       -1     
==========================================
- Hits         4061     4060       -1     
  Misses        361      361              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a673a93...847238a. Read the comment docs.

Thanks for this!

Nevermind, had to remove the branch argument in the workflow to make vercel publish work

I am running into the same problem. Is there any workaround?

Any workaround for making it work with datasette-publish-vercel. Currently getting below error

module initialization error: init() got an unexpected keyword argument 'config'
module initialization error
init() got an unexpected keyword argument 'config'

Codecov Report

Merging #1485 (be9aed9) into main (0d5cc20) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1485   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4422     4422           
=======================================
  Hits         4061     4061           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d5cc20...be9aed9. Read the comment docs.

I'll open a separate issue for removing _next= when running a search.

I think that SQLite error message difference was caused by https://github.com/python/cpython/commit/a50e28377bcf37121b55c2de70d95a5386c478f8 or related work.

I applied my PR against Janus to my local copy of Datasette like so:

pip uninstall janus
pip install https://github.com/aio-libs/janus/archive/9e13d3fb74e2c93d7501443b370a455d1b302b1f.zip

Then I ran the Datasette tests and got a much happier pass rate.

I submitted a PR to Janus with a workaround for this: https://github.com/aio-libs/janus/pull/359

There's a tiny chance this could be a bug in Python 3.10 itself - I filed an issue here: https://bugs.python.org/issue45416 - in which I said:

In Python 3.10 it is not possible to instantiate an asyncio.Condition that wraps an asyncio.Lock without raising a "loop argument must agree with lock" exception.

This code raises that exception:

asyncio.Condition(asyncio.Lock())

This worked in previous Python versions.

Note that the error only occurs if an event loop is running. Here's a simple script that replicates the problem:

import asyncio

# This runs without an exception:
print(asyncio.Condition(asyncio.Lock()))

# This does not work:
async def example():
    print(asyncio.Condition(asyncio.Lock()))

# This raises "ValueError: loop argument must agree with lock":
asyncio.run(example())

And here's the relevant Janus code: https://github.com/aio-libs/janus/blob/d7970f8b76bcac2e087067ca4575ac845e481874/janus/__init__.py#L24-L42

class Queue(Generic[T]):
    def __init__(self, maxsize: int = 0) -> None:
        self._loop = current_loop()
        self._maxsize = maxsize

        self._init(maxsize)

        self._unfinished_tasks = 0

        self._sync_mutex = threading.Lock()
        self._sync_not_empty = threading.Condition(self._sync_mutex)
        self._sync_not_full = threading.Condition(self._sync_mutex)
        self._all_tasks_done = threading.Condition(self._sync_mutex)

        self._async_mutex = asyncio.Lock()
        # "loop argument must agree with lock" exception is raised here:
        self._async_not_empty = asyncio.Condition(self._async_mutex)
        self._async_not_full = asyncio.Condition(self._async_mutex)
        self._finished = asyncio.Event()
        self._finished.set()

Here's the code that raises that error: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/asyncio/locks.py#L219-L234

class Condition(_ContextManagerMixin, mixins._LoopBoundMixin):
    """Asynchronous equivalent to threading.Condition.
    This class implements condition variable objects. A condition variable
    allows one or more coroutines to wait until they are notified by another
    coroutine.
    A new Lock object is created and used as the underlying lock.
    """

    def __init__(self, lock=None, *, loop=mixins._marker):
        super().__init__(loop=loop)
        if lock is None:
            lock = Lock()
        elif lock._loop is not self._get_loop():
            raise ValueError("loop argument must agree with lock")

Only mention I can find of that "loop argument must agree with lock" error is here - which doesn't have any tips for a workaround yet: https://giters.com/django/channels_redis/issues/278

So maybe this is an issue with Janus? I'm using https://pypi.org/project/janus/ 0.6.1 which is the latest release, from October 2020.

Running pytest -x --pdb helped me see this error:

  File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 122, in dispatch_request
    await self.ds.refresh_schemas()
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 344, in refresh_schemas
    await self._refresh_schemas()
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 349, in _refresh_schemas
    await init_internal_db(internal_db)
  File "/Users/simon/Dropbox/Development/datasette/datasette/utils/internal_db.py", line 5, in init_internal_db
    await db.execute_write(
  File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 102, in execute_write
    return await self.execute_write_fn(_inner, block=block)
  File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 113, in execute_write_fn
    reply_queue = janus.Queue()
  File "/Users/simon/.local/share/virtualenvs/py310-Z8fTATkJ/lib/python3.10/site-packages/janus/__init__.py", line 39, in __init__
    self._async_not_empty = asyncio.Condition(self._async_mutex)
  File "/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/asyncio/locks.py", line 234, in __init__
    raise ValueError("loop argument must agree with lock")
ValueError: loop argument must agree with lock

Then I created myself a temporary 3.10 environment using pipenv like so:

cd /tmp
mkdir py310
cd py310
pipenv shell --python /Users/simon/.pyenv/versions/3.10.0/bin/python

And used that with my Datasette checkout like so:

cd ~/.../datasette
pip install -e '.[test]'
pytest

Thanks for the reply @simonw. What services have you had better success with than Cloud Run for larger database?

Also, what about my issue description makes you think there may be a workaround?

Is there any instrumentation I could add to see at which point in the deploy the memory usage spikes? Should I be able to see this whether it's running under Docker locally, or do you suspect this is Cloud Run-specific?

The 3.10 tests failed a lot. Trying to run this locally:

/tmp % pyenv install 3.10
python-build: definition not found: 3.10

The following versions contain `3.10' in the name:
  3.10.0a6
  3.10-dev
  miniconda-3.10.1
  miniconda3-3.10.1

See all available versions with `pyenv install --list'.

If the version you need is missing, try upgrading pyenv:

  brew update && brew upgrade pyenv

So trying:

brew update && brew upgrade pyenv

Then did this:

/tmp % brew upgrade pyenv 
==> Upgrading 1 outdated package:
pyenv 1.2.24.1 -> 2.1.0

This decided to upgrade everything by downloaded everything on the internet. Aah, Homebrew.

But it looks like I have 3.10.0 available to pyenv now.

/tmp % pyenv install 3.10.0
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.0.tar.xz...
-> https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz
Installing Python-3.10.0...
...

Codecov Report

Merging #1481 (18a4907) into main (6388617) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1481   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6388617...18a4907. Read the comment docs.

I've had this problem too - my solution was to not use Cloud Run for databases larger than about 2GB, but the way you describe it here makes me think that maybe there is a workaround here which could get it to work.

On further investigation this isn't related to _search at all - it happens when you explicitly sort by _sort=rowid and apply a _next

• https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 works without an error (currently)
• https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid shows that error

Here's the exception:

-> params[f"p{len(params)}"] = components[0]
(Pdb) list
603     
604                     # Figure out the SQL for next-based-on-primary-key first
605                     next_by_pk_clauses = []
606                     if use_rowid:
607                         next_by_pk_clauses.append(f"rowid > :p{len(params)}")
608  ->                     params[f"p{len(params)}"] = components[0]
609                     else:
610                         # Apply the tie-breaker based on primary keys
611                         if len(components) == len(pks):
612                             param_len = len(params)
613                             next_by_pk_clauses.append(

Debugger shows that components is an empty array, so components[0] cannot be resolved:

-> params[f"p{len(params)}"] = components[0]
(Pdb) params
{'search': 'hello'}
(Pdb) components
[]

So the bug is in this code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/views/table.py#L604-L617

As dogsheep-photos already uses osxphotos to load photos you can access the EXIF data via osxphotos. Apple Photos imports a small subset of EXIF data at the time the photo is imported and osxphotos provides this via the exif_info property. If you want the full EXIF data, osxphotos also provides a wrapper around exiftool.

Maybe the exif-loader from the SpatiaLite project could be useful as a reference even it is written in C and it also saves images as blobs https://www.gaia-gis.it/fossil/spatialite-tools/file?name=exif_loader.c&ci=tip. The tool is also integrated into the spatialite-gui application.

I found some user documentation from the web archive http://web.archive.org/web/20180629041238/https://www.gaia-gis.it/spatialite-2.3.1/spatialite-exif-2.3.1.html.

Actually no, from that stack trace you provided:

  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py", line 283, in package
    call(args)
  File "c:\users\grott\anaconda3\lib\contextlib.py", line 119, in __exit__
    next(self.gen)
  File "c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py", line 451, in temporary_docker_directory
    tmp.cleanup()

It looks like the problem occurs here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L449-L452

I suspect the root cause of this may be in this code: https://github.com/simonw/datasette/blob/63886178a649586b403966a27a45881709d2b868/datasette/utils/__init__.py#L673-L677

I'm pretty sure this is a Windows issue, not a Fly issue. I imagine it affects other forms of datasette publish too.

Thanks, but this one has a different error type. Unfortunately, still not working.

My search yielded these four entries: https://github.com/simonw/datasette/issues?q=PermissionError%3A+%5BWinError+32%5D+

Maybe this is the closet hit?
https://github.com/simonw/datasette/issues/744

@simonw any feedback/thoughts?

Hi @simonw - wondering if you might be able to shed some light here. I've seemed to reproduce this issue.
Here's the stacktrace:

...     db["potholes"].insert(pothole, pk='id', alter=True, replace=True)
... 

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2481, in insert
    return self.insert_all(
  File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2596, in insert_all
    self.insert_chunk(
  File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2424, in insert_chunk
    row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
IndexError: list index out of range

Interesting enough, I found that omitting the pk param does not throw the error.

Let me know how I can help out!

Similar trouble with ageinfo using 0.22. Here's what my ageinfo.js file looks like:

window.YTD.ageinfo.part0 = [
  {
    "ageMeta" : { }
  }
]

Commenting out the registration for ageinfo in archive.py gets my archive to import.

I could even have those replacement characters be properties of the Database class, so uses can sub-class and change them.

class Database:
    left_brace_replace = "_"
    right_brace_replace = "_"
    ...

I think I like the underscore option best. I don't like the idea of injecting surprise ( ) parenthesis, and having them vanish entirely could result in things like item[price] becoming itemprice which feels confusing.

item_price_ is a little ugly but I think I can live with it.

What are my options for replacing those characters?

• [ becomes ( and ] becomes )
• [ and ] are removed entirely
• [ and ] both become _

I'm inclined to just fix them and not have an option for opting-out of fixing them, since it adds quite a bit of cruft to the overall API design for an option that maybe no-one will ever use.

If I add a new parameter for opting in and out of fixing these, what should it be called? A few options:

• fix_columns=False
• rename_bad_columns=False
• fix_column_names=False
• error_on_invalid_column_names=True

Here's the code where this happens: https://github.com/simonw/sqlite-utils/blob/54191d4dc114d7dc21e849b48a4d5ae4f9e601ca/sqlite_utils/db.py#L2943-L2948

It's called from three different methods in db.py: create_table_sql(), update() and insert_all().

So either I do the automatic replacement, or I let the user request automatic replacement, or a third option: I do automatic replacement but let the user opt to receive an error instead.

Quoting https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856 in full:

I'm not sure what to do about this one.

I can't fix it: this bug in Python's sqlite3 module means that even if I write a database out with column names that include [] I won't be able to read them back again.

So... I could do one of the following:

• Throw an error if a column name includes those characters. That's my preferred option I think.
• Automatically replace [ in column names with ( and ] with )
• Do the automatic replacement but show a user-visible warning when I do it
• Throw an error, but give the user an option to run with e.g. --fix-column-names which applies that automatic fix.

Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.

Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-directly-against-csv-or-json

If two files have the same name they will be assigned a numeric suffix:

$ sqlite-utils memory foo/data.csv bar/data.csv "select * from data_2"

Here's the relevant code: https://github.com/simonw/sqlite-utils/blob/7427a9137f60de961b6331d0922a3f03da0d1890/sqlite_utils/cli.py#L1289-L1292

I can fix this by checking to see if csv_table is already in use and adding a suffix.

Oddly I can't replicate this on macOS:

(sqlite-utils) sqlite-utils % ls foo/*.csv
foo/bug.csv
(sqlite-utils) sqlite-utils % ls bar/*.csv
bar/bug.csv
(sqlite-utils) sqlite-utils % sqlite-utils memory foo/bug.csv bar/bug.csv --schema
CREATE TABLE "bug" (
   [col1] TEXT,
   [col2] TEXT
);
CREATE VIEW t1 AS select * from [bug];
CREATE VIEW t AS select * from [bug];

Wow, that was fast! Thank you!

The t1 and t2 aliases were meant to handle this case, but the are no good if the tool throws an error.

The bug is in this code: https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/sqlite_utils/cli.py#L2205-L2227

Good catch, thanks.

Side-note: Django 4.0 will switch from using pytz to using the standard library zoneinfo module introduced in Python 3.9, which has a backport that works as far back as 3.6: https://github.com/pganssle/zoneinfo (https://pypi.org/project/backports.zoneinfo/)

If I need to handle timezones I'll use that, but I think I can get away without it?

Django does this: https://github.com/django/django/blob/b0ed619303d2fb723330ca9efa3acf23d49f1d19/setup.cfg#L39-L43

install_requires =
    asgiref >= 3.3.2
    backports.zoneinfo; python_version<"3.9"
    sqlparse >= 0.2.2
    tzdata; sys_platform == 'win32'

The audiences I care about here are:

• Producers of this timestamp - generally that will be users who are using datasette publish to share their data
• Human consumers of this timestamp - end users who look at a Datasette site and want to know how recent the data is
• Machine consumers of this timestamp - API integrations that might want to check if a Datasette instance has been updated before downloading new data

For producers I think there are going to be two categories. The first is users who run "publish" and want the site to reflect when they did so (probably using --updated=now when they publish). The second are users who are willing to spend more time thinking about this - for example my various git scraping projects where I want to use a date derived from the git history.

For humans... I'd like to be able to calculate a relative time (3 hours ago) in addition to showing the display time, because that helps avoid confusion over timezones.

For machine consumers, it might be nice to offer the option of a calculated Unix timestamp-since-1970, since those can be easier to work with in some languages than running a full ISO date parser.

The simplest possible version of this: it's always represented as a UTC ISO datetime, like this:

"updated": "2020-10-31T12:00:00+00:00"

Later versions of Datasette could extend this to handle other timezones or support just the date (though that's a backwards incompatible change so probably better to decide on the date thing right now).

So this is a metadata key called updated which can be applied at the table, database or instance level. It is represented as a .isoformat() timestamp.

Question: should I support just the date - yyyy-mm-dd - in addition to the datetime?

I think so. I can easily imagine situations where the exact time of day that a change was made hasn't been recorded, but the overall date is known.

But in that case, should the updated key sometimes be yyyy-mm-dd and sometimes be the full isoformat datetime? Or should there be an updated_date key that's used for just the date?

I'm going to use https://github.com/dateutil/dateutil for this - it's been maintained constantly (by an evolving team of contributors) since 2003 and is a very trustworthy dependency.

Thanks for this!

Codecov Report

Merging #1476 (2eb01ff) into main (b28b6cd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1476   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b28b6cd...2eb01ff. Read the comment docs.

Still a good idea today too! Would be great for https://cdc-vaccination-history.datasette.io/ for example.

I replaced:

<a class="text-3xl font-bold leading-none" href="https://www.saferortoxic.org">
<img 
src="https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp" 
class="custom-logo" 
alt="Logo: Safer or Toxic?" 
width="300px">
</a>

with:

<img src="https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp" 
alt="Logo: Safer or Toxic?" 
style="width:300px; cursor:pointer;" 
onclick="window.location.href='https://saferortoxic.org';return false;">

I'd still love to know what caused this (and how to troubleshoot to figure it out), so I'll leave it open for a bit, but I do have a functional logo linking to the Hugo home page, at least locally. I'll likely push tomorrow.

(Before trying this, I tried to apply a background image to the a tag. That didn't work.)

An update, if I remove the img tag and replace it with the text, "Safer or Toxic?" it links to the right place.

Also, if I keep things exactly as they are, and it improperly, but consistently goes to the undefined page, on THAT 404 page, a click on the image properly clicks through to the www.SaferOrToxic.org page.

Weird stuff.

That's so useful @sethvincent! Really interesting reading your code there, especially clever how you're using the base_url config.

I'd be very interested to see what your demo looks like without using serverless - completely agree that the less additional dependencies there are for this the better.

I'm also very interested in figuring out a way to run Datasette in Lambda but with the SQLite database on an EFS volume. Do you have a feel for how hard that would be?

:wave: I just put together a small example using the lambda container image support: https://github.com/sethvincent/datasette-aws-lambda-example

It uses mangum and AWS's python runtime interface client to handle the lambda event stuff.

I'd be happy to help with a publish plugin for AWS lambda as I plan to use this for upcoming projects.

The example uses the serverless cli for deployment but there might be a more suitable deployment approach for the plugin. It would be cool if users didn't have to install anything additional other than the aws cli and its associated config/credentials setup.

Superseded by #1471.

haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?

Not certain, but I think tests in CI run on Ubuntu and don't appear to install any additional Sqlite-related dependencies, and so my guess is the version of Sqlite installed by default on Ubuntu has the SQLITE_ENABLE_FTS3_PARENTHESIS option enabled and so doesn't run into this issue.

Codecov Report

Merging #1385 (db78094) into main (ea627ba) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head db78094 differs from pull request most recent head 8d78c8c. Consider uploading reports for the commit 8d78c8c to get more accurate results

@@           Coverage Diff           @@
##             main    #1385   +/-   ##
=======================================
  Coverage   91.70%   91.70%           
=======================================
  Files          34       34           
  Lines        4364     4364           
=======================================
  Hits         4002     4002           
  Misses        362      362           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea627ba...8d78c8c. Read the comment docs.

Especially relevant now that 0.2.0 is out which is a much higher quality release. https://github.com/simonw/datasette-app/releases/tag/0.2.0

Here's a already open issue for this: #972

At the moment this isn't possible - though there's a workaround which is to define a SQL view for the query, at which point facets will be displayed again.

I did a lot of the work required to support this when I refactored how facets worked a while back - but to finally implement this I need to refactor the table view and the arbitrary query view to share much more logic than they do at the moment.

Here's the code at fault: https://github.com/simonw/datasette/blob/b28b6cd2fe97f7e193a235877abeec2c8eb0a821/datasette/static/table.js#L137-L146

haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?

dupe of #293?

Thanks, that does look similar!

Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.

Ah that makes sense, I guess that's why this isn't just always installed already. I wonder if a possible solution to this issue could be doing feature detection on whether this feature is supported by the current Sqlite version, and if not these tests could be disabled locally? But possibly there's a better way to handle this, will see what @simonw thinks

Codecov Report

Merging #326 (078a087) into main (77c240d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #326   +/-   ##
=======================================
  Coverage   96.58%   96.58%           
=======================================
  Files           5        5           
  Lines        2223     2223           
=======================================
  Hits         2147     2147           
  Misses         76       76           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77c240d...078a087. Read the comment docs.

thanks for the response! full errors attached; excerpt:

...


    def test_searchmode(table_metadata, querystring, expected_rows):
        with make_app_client(
            metadata={"databases": {"fixtures": {"tables": {"searchable": table_metadata}}}}
        ) as client:
            response = client.get("/fixtures/searchable.json?" + querystring)
>           assert expected_rows == response.json["rows"]
E           AssertionError: assert [[1, 'barry c...sel', 'puma']] == []
E             Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']
E             Use -v to get the full diff

/Users/t/dev/datasette/tests/test_api.py:1115: AssertionError

errors.txt

A quick scan of #1223 suggests you're right. Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.

What are the full errors you're getting?

This may be the same issue as described in https://github.com/simonw/datasette/pull/1223 - essentially the test suite (and corresponding Datasette features I assume) are by default implicitly dependent on your Sqlite installation having been compiled with the SQLITE_ENABLE_FTS3_PARENTHESIS option. If this is the same issue then I think this can be fixed either by recompiling with that option or (probably more easily) by running pip install pysqlite3-binary, which will be used in preference to your system Sqlite installation and has this option enabled.

on commit d57ab156b35ec642

I think this is a setting. There are two relevant settings at the moment:

    "template_debug": false,
    "trace_debug": false,

For consistence then this should be called something_debug - but do I want a single setting that exposes the _internal database and adds those debug options to the menu, or do I want those as two separate settings?

• internal_debug to enable access to that _internal database
• menu_debug for those menu options?

I don't think I'll adopt it for this project. For example, here:

     response = Response.redirect("/")
-    response.set_cookie("ds_actor", datasette.sign({
-        "a": {
-            "id": "cleopaws"
-        }
-    }, "actor"))
+    response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))

I chose to use the multi-line version to help emphasize the structure - the single-line replacement loses that. I think I'll continue to make my own editorial choices about how the code examples are laid out.

Here's the diff it produced from that first run:
`diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signedds_actor`` cookies themselves like so:
.. code-block:: python

 response = Response.redirect("/")

• response.set_cookie("ds_actor", datasette.sign({
• "a": {
• "id": "cleopaws"
• }
• }, "actor"))
• response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor"))

Note that you need to pass "actor" as the namespace to :ref:datasette_sign.

@@ -412,12 +408,16 @@ To include an expiry, add a "e" key to the cookie value containing a `base62
expires_at = int(time.time()) + (24 * 60 * 60)

 response = Response.redirect("/")

• response.set_cookie("ds_actor", datasette.sign({
• "a": {
• "id": "cleopaws"
• },
• "e": baseconv.base62.encode(expires_at),
• }, "actor"))
• response.set_cookie(
• "ds_actor",
• datasette.sign(
• {
• "a": {"id": "cleopaws"},
• "e": baseconv.base62.encode(expires_at),
• },
• "actor",
• ),
• )

The resulting cookie will encode data that looks something like this:

diff --git a/docs/spatialite.rst b/docs/spatialite.rst
index d1b300b..556bad8 100644
--- a/docs/spatialite.rst
+++ b/docs/spatialite.rst
@@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns,
.. code-block:: python

 import sqlite3

• conn = sqlite3.connect('museums.db')
  +
• conn = sqlite3.connect("museums.db")
  # Lead the spatialite extension:
  conn.enable_load_extension(True)
• conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
• conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
  # Initialize spatial metadata for this database:
• conn.execute('select InitSpatialMetadata(1)')
• conn.execute("select InitSpatialMetadata(1)")
  # Add a geometry column called point_geom to our museums table:
  conn.execute("SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);")
  # Now update that geometry column with the lat/lon points
• conn.execute('''
• conn.execute(
• """
  UPDATE museums SET
  point_geom = GeomFromText('POINT('||"longitude"||' '||"latitude"||')',4326);
• ''')
• """

• )
  # Now add a spatial index to that column
  conn.execute('select CreateSpatialIndex("museums", "point_geom");')
  # If you don't commit your changes will not be persisted:
  @@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
  .. code-block:: python

  import sqlite3
  - conn = sqlite3.connect('places.db')
  +
  + conn = sqlite3.connect("places.db")
  # Enable SpatialLite extension
  conn.enable_load_extension(True)
  - conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
  + conn.load_extension("/usr/local/lib/mod_spatialite.dylib")
  # Create the masic countries table
  - conn.execute('select InitSpatialMetadata(1)')
  - conn.execute('create table places (id integer primary key, name text);')
  + conn.execute("select InitSpatialMetadata(1)")
  + conn.execute("create table places (id integer primary key, name text);")
  # Add a MULTIPOLYGON Geometry column
  conn.execute("SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);")
  # Add a spatial index against the new column
  @@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
  from shapely.geometry.multipolygon import MultiPolygon
  from shapely.geometry import shape
  import requests
  - geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json()
  +
  + geojson = requests.get(
  + "https://data.whosonfirst.org/404/227/475/404227475.geojson"
  + ).json()
  # Convert to "Well Known Text" format
  - wkt = shape(geojson['geometry']).wkt
  + wkt = shape(geojson["geometry"]).wkt
  # Insert and commit the record
  - conn.execute("INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))", (
  - "Wales", wkt
  - ))
  + conn.execute(
  + "INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))",
  + ("Wales", wkt),
  + )
  conn.commit()

Querying polygons using within()
diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst
index bd60a4b..5af01f6 100644
--- a/docs/writing_plugins.rst
+++ b/docs/writing_plugins.rst
@@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a my_plugin.py file

 from datasette import hookimpl

+
@hookimpl
def prepare_connection(conn):
- conn.create_function('hello_world', 0, lambda: 'Hello world!')
+ conn.create_function("hello_world", 0, lambda: "Hello world!")

If you save this in plugins/my_plugin.py you can then start Datasette like this::

@@ -60,22 +61,18 @@ The example consists of two files: a setup.py file that defines the plugin:

 from setuptools import setup

• VERSION = '0.1'

• VERSION = "0.1"

  setup(
  - name='datasette-plugin-demos',
  - description='Examples of plugins for Datasette',
  - author='Simon Willison',
  - url='https://github.com/simonw/datasette-plugin-demos',
  - license='Apache License, Version 2.0',
  + name="datasette-plugin-demos",
  + description="Examples of plugins for Datasette",
  + author="Simon Willison",
  + url="https://github.com/simonw/datasette-plugin-demos",
  + license="Apache License, Version 2.0",
  version=VERSION,
  - py_modules=['datasette_plugin_demos'],
  - entry_points={
  - 'datasette': [
  - 'plugin_demos = datasette_plugin_demos'
  - ]
  - },
  - install_requires=['datasette']
  + py_modules=["datasette_plugin_demos"],
  + entry_points={"datasette": ["plugin_demos = datasette_plugin_demos"]},
  + install_requires=["datasette"],
  )

And a Python module file, datasette_plugin_demos.py, that implements the plugin:
@@ -88,12 +85,12 @@ And a Python module file, datasette_plugin_demos.py, that implements the plu

 @hookimpl
 def prepare_jinja2_environment(env):

• env.filters['uppercase'] = lambda u: u.upper()

• env.filters["uppercase"] = lambda u: u.upper()

  @hookimpl
  def prepare_connection(conn):
  - conn.create_function('random_integer', 2, random.randint)
  + conn.create_function("random_integer", 2, random.randint)

Having built a plugin in this way you can turn it into an installable package using the following command::
@@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI

.. code-block:: python

• package_data={
• 'datasette_plugin_name': [
• 'static/plugin.js',
• ],
• },
• package_data = (
• {
• "datasette_plugin_name": [
• "static/plugin.js",
• ],
• },
• )

Where datasette_plugin_name is the name of the plugin package (note that it uses underscores, not hyphens) and static/plugin.js is the path within that package to the static file.

@@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same package_data mec

.. code-block:: python

• package_data={
• 'datasette_plugin_name': [
• 'templates/my_template.html',
• ],
• },
• package_data = (
• {
• "datasette_plugin_name": [
• "templates/my_template.html",
• ],
• },
• )

You can also use wildcards here such as templates/*.html. See datasette-edit-schema <https://github.com/simonw/datasette-edit-schema>__ for an example of this pattern.
```

(datasette) datasette % blacken-docs docs/*.rst
docs/authentication.rst: Rewriting...
docs/internals.rst:169: code block parse error Cannot parse: 14:0: <line number missing in source>
docs/plugin_hooks.rst:251: code block parse error Cannot parse: 6:4:     ]
docs/plugin_hooks.rst:312: code block parse error Cannot parse: 38:0: <line number missing in source>
docs/spatialite.rst: Rewriting...
docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: <line number missing in source>
docs/writing_plugins.rst: Rewriting...

This idea makes sense to me.

There's actually an existing option that takes a path, called --get - it returns the HTML or JSON for that oath directly to the console, eg datasette my.db --get /mydb/mytable.json

So... one option would be to allow combining that with -o to open that URL in the browser:

datasette my.db -o --get /mydb

So some options here are:

• datasette my.db --open-url /mydb
• datasette my.db --open-path /mydb
• datasette my.db --open --get /mydb

I quite like that last combination option, mainly to avoid adding even more command options.

I'll add researchers too.

Great idea!

I love it! maybe 'researchers' instead? Or 'scientists and researchers'?

Additionally, assuming the line numbers match up with the provided enenx file, the mentioned line plus one before and after is as follows:

<![CDATA[>]]>
</span></div>
<div style="padding: 0px; font-family: Arial, sans-serif; font-size: 12px; line-height: 16px; white-space: pre-wrap;"><br style=" padding: 0px;"/>

Found the cause, it's the other commands. PR #59 submitted.

I used the sticky footer mechanism in datasette.app: https://github.com/simonw/datasette.app/issues/3

Codecov Report

Merging #1453 (4f492a7) into main (67cbf0a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1453   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67cbf0a...4f492a7. Read the comment docs.