Activity for Isync
-
Oswald Buddenhagen modified ticket #91
mbsync: "Warning: SASL wants more steps despite successful IMAP authentication. Ignoring..."
-
Space created ticket #91
mbsync: "Warning: SASL wants more steps despite successful IMAP authentication. Ignoring..."
-
Stephan Gerth posted a comment on ticket #90
FWIW: Source server is Dovecot 2.3.13 (Debian 11), target server is Dovecot 2.4.1-4 (Debian 13).
-
Connection loss for IMAP2IMAP sync on large mailboxes
-
ok, thanks, something is obviously going wrong. first it queues 22 fetches (which is incompatible with a pipeline depth of 1), then it queries the buffer size for all remaining messages (which is zero, because no appends have been scheduled), then it queues all remaining fetches. then the first fetches are completing, upon which it correctly schedules the first append, and the next one only once the first one completes (this is how pipelinedepth 1 should work). one notable observation is that the...
-
I've sent you a log of a -Ddn run with BufferLimit 1, PipelineDepth 1 on the target and FSync no to the maintainer address.
-
the only legitimate way how that can happen is that the IMAP commands were actually all already sent out, and the queuing is happening on the target server. -Ddn should confirm that. the way to work around that would be limiting PipelineDepth to something reasonable, say 50.
-
I think the disk setup is not the actual problem since the network connection could always be the main bottleneck (imagine syncing over a 50MBit internet connection). But I think I figured out why the BufferLimit has no effect (lines refer latest git 42cfa1): Line sync.c:1468 checks via get_memory_usage if BufferLimit has been already reached or overrun and handles that. However, according to the logs all calls to get_memory_usage returned no usage at all: N: Called get_memory_usage, ret=0 This explains...
-
the fs load result is surprising. what are the actual numbers? network bw, disk bw, fs type & mount options, etc. that effect of bufferlimit sounds just wrong. please mail me the log.
-
I had a look at iotop and I could see that the write rate was 1,5x compared the Fsync yes setting. So it seems to make a difference but does not increase write rate enough. Setting BufferLimit 1 pushes message storage further to the end in the log file (-Dd) but it still fetches all messages so that the connection to the source server runs into the timeout (while it is working on storing all mails into the target mailbox folder (about 120k lines only enter/leave store_msg lines).
-
Setting FSYNC no did not make any difference. weird. have a look at iotop on the target server with the two settings. also check the mount options of the volume. it's kinda unexpected that the process is disk-bound to such a degree. Would it make sense to set it to some smaller value than the default value so that it cannot fetch the whole mailbox folder [...]? yes. you could actually use something ridiculously small (even 1) to avoid that multiple STOREs are queued up at all.
-
I've tried with the -Dd option: The output makes sense so far: A lot of interleaved messages of such as this snippet (within the large mailbox folder already): F: [ 120772] Callback enter fetch_msg, sts=0, flags=, date=1789297202, size=110049 N: [ 228630] Enter store_msg, flags=, date=1789297202, size=110071, to_trash=no N: [ 228630] Leave store_msg F: [ 120772] Callback leave fetch_msg F: [ 120773] Callback enter fetch_msg, sts=0, flags=, date=1789279202, size=122022 N: [ 228631] Enter store_msg,...
-
there is no such mechanism; i just didn't anticipate this situation. assuming you are using the default BufferLimit, the timeout cannot possibly occur while messages are being transferred. (*) so it must be during message enumeration. but that should be also on the order of 10 MB with your mailbox size, so i dunno. have you tried the -Dd log? if you can't make sense of it, you can mail it to me privately. EDIT: (*) well, actually, if the target server is using a legacy fs that does fsync inefficiently,...
-
there is no such mechanism, i just didn't anticipate this situation. assuming you are using the default BufferLimit, the timeout cannot possibly occur while messages are being transferred. so it must be during message enumeration. but that should be also on the order of 10 MB with your mailbox size, so i dunno. have you tried the -Dd log? if you can't make sense of it, you can mail it to me privately.
-
Regarding the working of isync this was just guessing from what I've observed, sorry! The mailbox folder in question has about 108k emails, so that's far from millions of messages. I've tried to speed-up the target server by putting the target mailbox on a ramdisk. This lets the synchronization finish before the timeout hits which solves it in my case. However, for larger mailboxes or slower servers the issue would surely occur again. How is the general mechanism working when target server takes...
-
that's just not how isync works. you can watch it by adding -Dd. -Dn (or just -D) will produce even more output, but based on the info so far i expect that to be somewhat useless due to the sheer volume of repetetive content. how large is "quite large"? at normal speeds, such a duration would translate to millions of messages.
-
No, this happens somewhere in the middle (on the 43rd folder of 48 to be specific which happens to be quite large). I watched some numbers this time while running the synchronization: fetching from the source seems to be faster than pushing to the target mbsync seems to cache the whole(?) mailbox folder (memory usage went to max. 10G) when it tries to push the remaining mails to the target while the source mails have been already cached, the source connection is not seeing any activity During the...
-
at which point does that happen? while one side is still FETCHing the message list? for half an hour?!
-
Connection loss for IMAP2IMAP sync on large mailboxes
-
limit length of credentials supplied to LOGIN
-
Fails to build against OpenSSL4
-
Fix for OpenSSL4 ompatibility
-
mention that --dry-run isn't quite read-only
-
fix build with openssl 4
-
Finn Gärtner posted a comment on merge request #10
Here is the relevant source code bit at the OpenSSL_1_1_0 tag: https://github.com/openssl/openssl/blob/abd30777cc72029e8a44e4b67201cae8ed3d19c1/crypto/asn1/asn1_lib.c#L366
-
thanks, this looks reasonable. one thing: i'd like to see a somewhat definitive statement (with source) that the function is actually available since 1.1.
-
Fix for OpenSSL4 ompatibility
-
Fails to build against OpenSSL4
-
mention that --dry-run isn't quite read-only
-
flip inclusion hierarchy of generated proxy driver source
-
Replace use of deprecated SecKeychain API
-
Add no-fail Core Foundation wrappers
-
Segregate macOS-specific keychain code
-
Add IncludeCmd directive to config parser
-
Refactor printing configuration errors
-
add support for single-quoting to our printf implementation
-
add unit test for our printf implementation
-
Mark Diekhans posted a comment on ticket #15
This is Claude. I view Claude as a very junior programmer assistant. For design and architectural, it is a starting point than needs a lot more digging. It is a just lot faster than google with little wisdom. It is much better at simpler tasks More on the use case, I am one of maintainers of the emacs VM mail client package (https://gitlab.com/emacs-vm/vm/). One feature is that VM allows one to add arbitrary labels on message. Locally, it stores these in message headers. When syncing with an IMAP...
-
i'm not convinced that even the first sentence in that "AI" output is correct. source? and i'm not convinced that it makes sense the link gm-labels with keywords/flags, as they serve somewhat different purposes. regarding x-keywords, the issue is that it is slow, as it requires each file to be opened to read these extended flags. it's basically unusable without a header cache. hence the interoperability issue. a survey of the various approaches used by relevant muas would be interesting.
-
i'm not convinced that even the first sentence is correct. source? and i'm not convinced that it makes sense the link gm-labels with keywords/flags, as they serve somewhat different purposes. regarding x-keywords, the issue is that it is slow, as it requires each file to be opened to read these extended flags. it's basically unusable without a header cache. hence the interoperability issue. a survey of the various approaches used by relevant muas would be interesting.
-
I am trying to use mbsync and dovecot to implement a local cache to work around how slow gmail IMAP is. Not being sync local label from emacs VM with gmail is a drawback. So I worked with Claude Code to determine what would be involved in implementing this feature. Here is a proposal and options: https://github.com/diekhans/isync-isync/blob/gmail-labels/mbsync-gmail-design.org Thoughts?
-
Suppress unavoidable memrchr() warning
-
Replace use of deprecated SecKeychain API
-
Add no-fail Core Foundation wrappers
-
Segregate macOS-specific keychain code
-
Add IncludeCmd directive to config parser
-
Refactor printing configuration errors
-
add support for single-quoting to our printf implementation
-
add unit test for our printf implementation
-
change reference file for man page date stamps
-
include version.sh in tar-balls
-
Richard Körber created ticket #88
Wildcard cert not accepted
-
Failing to fetch complete mailbox from Yahoo!
-
the culprit is the server returning partial fetch responses, which are tagged with [LIMIT] . this obviously completely derails isync. yahoo is attempting to standardize (a slightly more advanced version of) this behavior in rfc9738. to cite from this awesome document: If the server has a MESSAGELIMIT of 1000, the client will only be able to download 1000 of the most recent messages; the client will not understand why, will not be prepared to recover from the situation, and will act as if it is interacting...
-
John Smith posted a comment on ticket #87
I've been unable to reproduce it using Dry Run. After several attempts, it always finishes with: Channels: 1 Boxes: 20 Far: +0 *0 #0 -0 Near: +122976 *0 #0 -0 Which looks like the correct grand total for the entire remote store (i.e inclusive of other folders, besides INBOX). So I haven't been able to reproduce the weird cap at 2000, or the message deletions. Some more context: I was originally using version 1.4.4-5. Over the course of a dozen or so attempts to synchronise the entire Yahoo! store...
-
this sounds bad. it seems that it thinks that the downloads were completed, so it treats the missing mails as deletions. and it somehow fails to see 70k+ messages. to debug this, i need the full log (with -D) of both the initial sync attempt and the destructive followup. that means that you need to start from scratch locally. do the second run with --dry-run, which should be good enough without causing actual damage. you can mail me the logs in private.
-
Failing to fetch complete mailbox from Yahoo!
-
Failing to fetch complete mailbox from Yahoo!
-
Derren Brown posted a comment on ticket #73
Zed editor with a deepseek chat model to be fair, it did a lot of capability and domain checks, i asked it to remove those
-
which coding agent exactly was that? the patch looks about correct, except that the actual ID string is mildly stupid. i'll have to check the spec for the correct fields and command order.
-
I Asked Coding Agent to fix this for me. here's what I got: it a/src/drv_imap.c b/src/drv_imap.c index 9b170ab..7b7f607 100644 --- a/src/drv_imap.c +++ b/src/drv_imap.c @@ -9,6 +9,7 @@ #include "imap_p.h" #include "socket.h" +#include "config.h" #include <ctype.h> #include <sys/wait.h> @@ -2088,6 +2089,8 @@ static void imap_open_store_authenticate_p3( imap_store_t *, imap_cmd_t *, int ) static void imap_open_store_authenticate2( imap_store_t * ); static void imap_open_store_authenticate2_p2( imap_store_t...
-
I Asked Coding Agent to fix this for me. here's what I got: it a/src/drv_imap.c b/src/drv_imap.c index 9b170ab..7b7f607 100644 --- a/src/drv_imap.c +++ b/src/drv_imap.c @@ -9,6 +9,7 @@ #include "imap_p.h" #include "socket.h" +#include "config.h" #include <ctype.h> #include <sys/wait.h> @@ -2088,6 +2089,8 @@ static void imap_open_store_authenticate_p3( imap_store_t *, imap_cmd_t *, int ) static void imap_open_store_authenticate2( imap_store_t * ); static void imap_open_store_authenticate2_p2( imap_store_t...
-
they only offer this, as @shanyi posted in the link(https://help.mail.163.com/faqDetail.do?code=d7a5dc8471cd0c0e8b4b8f4f8e49998b374173cfe9171305fa1ce630d7f67ac2eda07326646e6eb0): ********************代码块区域开始******************** Properties props = new Properties(); props.setProperty("mail.store.protocol", "imap"); props.setProperty("mail.imap.host", "imap.163.com"); props.setProperty("mail.imap.port", "143"); HashMap IAM = new HashMap(); //带上IMAP ID信息，由key和value组成，例如name，version，vendor，support-email等。...
-
126.com, 163.com and possibly 188.com. from the response I get, I don't think they have any plan to fix this though. I couldn't even get to the tech team
-
add support for imap id extension
-
so it appears that the various netease domains weren't fixed ....
-
I have written an email to kefu@188.com on this issue, In the mean time, would you consider implement this?
-
make imap_msgs test actually return non-zero on failure
-
clarify that maildir message moves should preserve the flag field
-
mention that plain --debug excludes driver-all as well
-
fix conditional on undefined value on some config syntax errors
-
fix crash in mkdir_p() with invalid path
-
please attach (or mail me) your config file and the logs of both runs, but with the -D option added. make sure that each run would actually have something to upgrade.
-
madroach created ticket #86
Upgrades do not work when synchronizing whole channel
-
Keith Bowes updated merge request #8
Update spec file
-
Use GNOME's secret-tool rather gnome-keyring-query
-
you may want to give the wip/includecmd branch a shot.
-
Kowalski posted a comment on ticket #17
something that has been requested exactly once so far I was looking for this too, and i'm sure many others were as well
-
Add IncludeCmd directive to config parser
-
Refactor printing configuration errors
-
add support for single-quoting to our printf implementation
-
add unit test for our printf implementation
-
make imap_msgs test actually return non-zero on failure
-
Add IncludeCmd directive to config parser
-
Add IncludeCmd directive to config parser
-
Refactor printing configuration errors
-
fix conditional on undefined value on some config syntax errors
-
W_Luis modified a comment on ticket #83
That was it. For some unknown reason I had an INBOX and an Inbox in the local machine, both empty. There was no counterpart in the mail server. I removed them and the warnings are gone. Furthermore, the summary line now says the expected number of boxes. Thanks!
-
That was it. For some unknown reason I had an INBOX and an Inbox in the local machine, both empty. There was no counterpart in the mail server. I removed them and the warnings are gone. Furthermore, the summary line now says the expected number of boxes.
-
the warning suggests that you have a literal "INBOX" folder in your maildir, next to the "Inbox" you actually want. you should delete it after making sure there is nothing worthwhile in it. the error is more mystifying, as this shouldn't be even possible with MapInbox. that might be a genuine bug. however, your summary line says 4 boxes, while your pattern specifies only 3, so something is inconsistent here.
-
This is my current mbsyncrc: Expunge Both Create Both MaildirStore emlocal Path ~/Maildir/ Inbox ~/Maildir/ MapInbox Inbox ImapStore em Host em User ... Pass ... CertificateFile /etc/ssl/certs/ca-certificates.crt TLSVersions +1.2 MapInbox Inbox Timeout 80 Channel emPriority Far :em: Near :emlocal: Pattern Inbox mbox Sent When I use it I get the message: $ mbsync emPriority Maildir warning: ignoring INBOX in /home/mochan/Maildir/ Error: channel emPriority: far side box INBOX cannot be opened anymore....
-
you should cut it down to what is actually needed. if some weirdness persists at this point, we can investigate.
-
possible test failures (linux)
-
dkwo2 posted a comment on ticket #85
Thanks a lot. If I cd src first, then all tests pass indeed.
-
make check does indeed run only the unit tests, while the integration test needs to be run manually. this has historical reasons, but given how slow the latter is, i am hesitant to change things. the test expects to be run from within the src directory with ./run-tests.pl.
-
possible test failures (linux) (v1.5.1)
-
Ben Vallack posted a comment on ticket #84
Many thanks.
-
I'm sorry; kind of short of time, so I returned to the configuration I showed above on 2025-01-26. The system has continued working but for the warnings Maildir warning: ignoring INBOX in /home/mochan/Maildir/ Error: channel emPriority: far side box INBOX cannot be opened. I haven't understood yet the situation, but my inboxes in the remote and local machine and my other mailboxes are being correctly synchronized.
-
I'm sorry; kind of short of time, so I returned to the configuration I showed above on 2025-01-26. The system has continued working but for the warnings Maildir warning: ignoring INBOX in /home/mochan/Maildir/ Error: channel emPriority: far side box INBOX cannot be opened.
-
Isync released /isync/1.5.1/isync-1.5.1.tar.gz
1 >
