issues

It looks like Swarm changed their schema and now both venues and checkins have createdAt fields.

Which breaks this view: https://github.com/dogsheep/swarm-to-sqlite/blob/719b6e96a016d0ca8b316d3bed9c2a7a0cb499ee/swarm_to_sqlite/utils.py#L171-L188

Just installed swarm-to-sqlite-0.3.2 and tried according to the docs:

Traceback (most recent call last): File "/usr/local/bin/swarm-to-sqlite", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/cli.py", line 73, in cli save_checkin(checkin, db) File "/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/utils.py", line 82, in save_checkin checkins_table.m2m("users", user, m2m_table="likes", pk="id") File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1914, in m2m id = other_table.insert(record, pk=pk, replace=True).last_pk File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1647, in insert return self.insert_all( File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1765, in insert_all self.insert_chunk( File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1575, in insert_chunk result = self.db.execute(query, params) File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 200, in execute return self.conn.execute(sql, parameters) sqlite3.OperationalError: table users has no column named lastName

While running swarm-to-sqlite it throws an error:

harper@:~/dogsheep/swarm$ swarm-to-sqlite checkins.db --save=checkins.json
Please provide your Foursquare OAuth token:
Importing 8127 checkins  [#################-------------------]   49%  00:01:52
Traceback (most recent call last):
File "/home/harper/.local/bin/swarm-to-sqlite", line 11, in <module>
    sys.exit(cli())
File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/home/harper/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
File "/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/cli.py", line 73, in cli
    save_checkin(checkin, db)
File "/home/harper/.local/lib/python3.6/site-packages/swarm_to_sqlite/utils.py", line 94, in save_checkin
    photos_table.insert(photo, replace=True)
File "/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py", line 963, in insert
    alter = self.value_or_default("alter", alter)
File "/home/harper/.local/lib/python3.6/site-packages/sqlite_utils/db.py", line 1142, in insert_all
    def upsert_all(
sqlite3.OperationalError: table photos has no column named hasSticker

Where should i dig in?

Fixes the errors: - sqlite3.OperationalError: table posts has no column named text - sqlite3.OperationalError: table photos has no column named hasSticker

That will cause sqlite-utils to notice if there's a missing column and add it. As recommended by @simonw

broke for me. very old checkins in 2010 had no source set.

The Foursquare checkins API supports "return every checkin occurring after this point" - I can pass it the maximum createdAt date currently stored in the database. This will allow for quick incremental fetches via a cron.

I will link to this from the documentation. See also this conversation on Twitter: https://twitter.com/simonw/status/1166822603023011840

I've decided to go with "copy and paste in a token" rather than hooking up a local web server that can have tokens passed to it.

Current test failure is due to timezone differences between my laptop and Circle CI:

https://circleci.com/gh/dogsheep/swarm-to-sqlite/3

E Full diff: E - [{'created': '2018-07-01T04:48:19', E ? ^ E + [{'created': '2018-07-01T02:48:19', E ? ^ E 'createdAt': 1530413299,

The timestamps I store in created should always be UTC.

This is a complement to the --load option - mainly useful for development purposes.

(I'll rename --file to --load as part of this issue).

The tool currently only works with a pre-prepared JSON file of checkins.

When called without options, it should prompt the user to paste in a Foursquare OAuth token.

The --token= option should work too, and should be backed up by an optional environment variable.